1# Changelog 2 3## v23.01: (Upcoming Release) 4 5### bdev 6 7Both of interleaved and separated metadata are now supported by the malloc bdev module. 8 9Protection information is now supported by the malloc bdev module. 10 11A new API `spkd_bdev_part_submit_request_ext` was added to specify a custom completion callback. 12 13### scheduler 14 15Changing scheduler from dynamic back to static is no longer possible, 16as there is no way of saving original SPDK thread distribution on reactors. 17 18### bdev 19 20New APIs `spdk_bdev_for_each_channel` and `spdk_bdev_for_each_channel_continue` and 21associated function pointers were added to iterate each channel of the required bdev. 22 23The RPC `bdev_get_iostat` now allows a user to query the per channel IO statistics for 24required bdev. 25 26New `spdk_bdev_copy_blocks` and `spdk_bdev_get_max_copy` APIs to support copy commands. 27 28A new API `spdk_bdev_io_get_submit_tsc` was added to get the submit_tsc of the bdev I/O. 29 30Bdevs will no longer have UUIDs generated based on timestamp and are responsible for 31setting this field themselves. Generation of UUIDs for NVMe bdevs may be enabled by 32running `bdev_nvme_set_options` RPC with `--generate-uuids` option. These identifiers 33are based on serial number and namespace ID and will always be the same for a given 34device. 35 36A new API `spdk_bdev_channel_get_histogram` was added to get the histogram of a specified 37channel for a bdev. 38 39### event 40 41Added core lock file mechanism to prevent the same CPU cores from being used by multiple 42SPDK application instances. 43 44Added a command line switch to disable CPU locks on SPDK startup. 45 46Added RPCs framework_enable_cpumask_locks and framework_disable_cpumask_locks to enable 47and disable CPU core locks in runtime. 48 49Added --rpcs-allowed command line option. Users can specify a comma-separated list of RPC 50names with this option to restrict allowed RPCs to only that list. 51 52### rpc 53 54Added spdk_rpc_set_allowlist to restrict allowed RPCs to the specified list. 55 56### bdevperf 57 58Promoted the application to example to match similar programs: fio_plugin and perf. 59It can now be found inside `examples/bdev/bdevperf`. 60 61### util 62 63New API `spdk_fd_group_get_epoll_event` that returns the epoll(7) event that 64caused a function callback in file descriptor group to execute. 65 66### json 67 68Added API `spdk_json_write_double` and `spdk_json_write_named_double` to allow 69for writing and decoding of the the double data type. 70 71### thread 72 73Added `spdk_thread_get_app_thread` which returns the first thread that was created using 74`spdk_thread_create`. 75 76Added `spdk_thread_is_running`. This returns `true` for a running thread, or `false` if 77its exit process has started using `spdk_thread_exit`. 78 79## v22.09 80 81### accel 82 83Many names were changed in the accel framework to make them consistent both with themselves and 84the rest of SPDK. The primary public header file is now named `include/spdk/accel.h`. 85 86Added a new runtime RPC `accel_get_opc_assignments` to get a list of current opcode to engine 87assignments. 88 89Added a new startup RPC `accel_assign_opc` to assign/override a specific opcode to 90an engine. 91 92### bdev 93 94New RPCs `bdev_xnvme_create` and `bdev_xnvme_delete` were added to support the xNVMe bdev. 95 96A new API `spdk_bdev_for_each_bdev_io` was added to execute the function on the appropriate 97thread for each bdev_io submitted to the bdev. 98 99A new API `spdk_bdev_get_current_qd` was added to measure and return the queue depth from a 100bdev. This API is available even when queue depth sampling is disabled. 101 102Added `spdk_bdev_seek_data`, `spdk_bdev_seek_hole` and `spdk_bdev_io_get_seek_offset` functions 103that start from a given offset and seek for next data or for next hole. At this time only 104supported by lvol bdev module. 105 106A new parameter `io_drain_timeout` has been added to `spdk_bdev` structure. It controls how long 107a bdev reset must wait for IO to complete prior to issuing a reset to the underlying device. 108If there is no outstanding IO at the end of that period, the reset is skipped. 109Best defined for bdevs that share an underlying bdev, such as multiple lvol bdevs sharing 110an nvme device, to avoid unnecessarily resetting the underlying bdev and affecting other 111bdevs that are sharing it. 112Modifying this field is optional. Setting the value to 0 keeps the original behavior 113to always send resets immediately, even if there is no I/O outstanding. 114 115### blobstore 116 117Reserve space for `used_clusters` bitmap. The reserved space could be used for blobstore growing 118in the future. 119 120Added `is_zeroes` operation to `spdk_bs_dev`. It allows to detect if logical blocks are backed 121by zeroes device and do a shortcut in copy-on-write flow by excluding copy part from zeroes device. 122 123Added `spdk_blob_get_next_allocated_io_unit` and `spdk_blob_get_next_unallocated_io_unit` functions 124that start from a given offset and seek for first io_unit belonging to an allocated cluster 125or first io_unit belonging to an unallocated cluster. 126 127Added `spdk_bs_grow` function to grow blobstore size if the underlying bdev size is increased. 128This is used by lvol bdev via `bdev_lvol_grow_lvstore` RPC. 129 130### daos 131 132Added new DAOS bdev module, that creates SPDK block device on top of DAOS DFS. 133Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_daos) for more details. 134 135### env 136 137Added `spdk_pci_register_device_provider` and matching `SPDK_PCI_REGISTER_DEVICE_PROVIDER` 138functions for registering PCI device providers. That allow the VMD driver to be notified 139when users want to attach a device under a given BDF and when a device is detached. 140 141### init 142 143`spdk_subsystem_init_from_json_config` now fails if the JSON configuration file is not 144an object with an array named "subsystems". 145 146### iscsi 147 148Added `bdev_iscsi_set_options` RPC to modify options of the iSCSI bdev module. 149 150### json 151 152Added `spdk_json_find` API return errcode: EPROTOTYPE - json not enclosed in {}. 153`spdk_json_find` now returns -EPROTOTYPE instead of -ENOENT if the object parameter 154does not point to a JSON object (i.e. is not enclosed with {}). 155 156### lvol 157 158Add `num_md_pages_per_cluster_ratio` parameter to the `bdev_lvol_create_lvstore` RPC. 159Calculate `num_md_pages` from `num_md_pages_per_cluster_ratio`, and pass it to `spdk_bs_opts`. 160 161### nvme 162 163Added SPDK_NVME_TRANSPORT_CUSTOM_FABRICS to enum `spdk_nvme_transport_type` to support custom 164fabric transport. SPDK_NVME_TRANSPORT_CUSTOM was intended to be non-fabric custom transport. 165 166Added a new function `spdk_nvme_ns_cmd_verify` to submit a Verify Command to a Namespace. 167 168Added `spdk_nvme_ctrlr_disable_read_changed_ns_list_log_page` to allow an application to 169tell the driver to not read the CHANGED_NS_LIST log page in response to a NS_ATTR_CHANGED 170AEN. When called the application is required to read this log page instead to clear the 171AEN. 172 173Added `psk` field to `spdk_nvme_ctrlr_opts` struct in order to enable SSL socket implementation 174of TCP connection and set the PSK. Applicable for TCP transport only. 175 176### raid 177 178Renamed the `raid5` module to `raid5f` to reflect that it is not a traditional 179RAID5 implementation - only full stripe writes are supported, partial stripe 180writes (read-modify-write) are not. 181 182### rpc 183 184Added `psk` parameter to `bdev_nvme_attach_controller` RPC in order to enable SSL socket implementation 185of TCP connection and set the PSK. Applicable for TCP transport only. 186 187New options `enable_ktls` and `tls_version` were added to the `sock_impl_set_options` structure. 188New options `psk_key` and `psk_identity` were added to the `sock_impl_set_options` structure. 189 190Added warning message for `bdev_rbd_create`, if it is used without -c. 191`bdev_rbd_create()` API without specifying -c is deprecated and will be removed in future release. 192 193Renamed `enable_vmd` RPC to `vmd_enable` to make it consistent with our naming scheme of 194`<subsystem>_<action>`. For now, the old name is still available, but is marked as deprecated. 195 196New function `spdk_rpc_get_method_state_mask` was added to RPC library. 197 198### sma 199 200Extended `VolumeParameters` with crypto parameters allowing the user to configure crypto when 201attaching a volume to a device. This interface is now supported by all upstream device types 202(nvmf-tcp, virtio-blk, vfio-user) using `bdev_crypto`. Users must specify the crypto engine to use 203under `crypto` section in config. It is also possible to register out-of-tree crypto engines by 204inheriting from the `CryptoEngine` class. 205 206Added two new methods: `SetQos` and `GetQosCapabilities` allowing the user to configure QoS on a 207per-device or per-volume level. Not all QoS settings have to be supported by each device, so users 208can use `GetQosCapabilities` to query them for that. All upstream device types support QoS on a 209per-volume level using bdev layer's QoS mechanism. 210 211### sock 212 213Added new `ssl` based socket implementation, the code is located in `module/sock/posix`. 214For now we are using hard-coded PSK and only support TLS 1.3. 215 216### util 217 218Added new functions: `spdk_hexlify` and `spdk_unhexlify`. 219 220A new API `spdk_xor_gen` was added to generate XOR from multiple source buffers. It is going to be 221used by `raid5f` for calculating parity. 222 223### vfu_tgt 224 225Added vfu_tgt library abstraction based on libvfio-user for PCI device emulation. 226Besides the NVMe device emulation, its now possible to emulate virtio-blk and virtio-scsi 227devices. 228 229### virtio 230 231virtio-vhost-user no longer tries to support dynamic memory allocation. The vhost target does 232not support the high rate of SET_MEM_TABLE vhost messages that result from dynamic memory 233allocation, so a virtio-vhost-user device will now present an ERRLOG, assert, and skip the 234SET_MEM_TABLE vhost message if a memory notification is received outside of the normal device 235start/stop. Applications using the virtio library in vhost-user mode should now pre-allocate 236the application's memory using the -s/--mem-size option and use single shared memory file 237segments using the -g/--single-file-segments option. 238 239Added `vfio-user` transport type for virtio-blk and virtio-scsi devices. 240 241### vmd 242 243Fixed hotplug when a device is inserted in a slot in which a disk was already enumerated previously. 244Added two new RPCs: `vmd_remove_device` simulating a hotremove, and `vmd_rescan`, which rescans all 245buses managed by the VMD driver and hotplugs all newfound devices. 246 247## v22.05 248 249### sock 250 251A new option `ack_timeout` was added to the `spdk_sock_opts` structure. 252 253### util 254 255A new parameter `bounce_iovcnt` was added to `spdk_dif_generate_copy` and `spdk_dif_verify_copy`. 256The `bounce_iovcnt` is used to specify the number of bounce_iov to support multiple block-aligned 257fragment copies. 258 259A new API `spdk_copy_iovs_to_buf` and `spdk_copy_buf_to_iovs` were added to copy iovs to buf or 260copy buf to iovs. There're many cases need to use these two APIs. 261 262### bdev 263 264Removed deprecated spdk_bdev_module_finish_done(). Use spdk_bdev_module_fini_done() instead. 265 266A new API `spdk_bdev_unregister_by_name` was added to handle race conditions correctly. 267 268New APIs, `spdk_for_each_bdev` and `spdk_for_each_bdev_leaf`, were added to provide iteration 269safe for race conditions. 270 271A new RPC `bdev_nvme_get_io_paths` was added to get all active I/O paths. 272 273A new RPC `bdev_nvme_set_preferred_path` was added to set preferred I/O path for an NVMe bdev 274when in multipath mode. This RPC does not support NVMe bdevs in failover mode. 275 276A new RPC `bdev_nvme_set_multipath_policy` was added to set multipath policy of a NVMe bdev 277in multipath mode. 278 279A new option `disable_auto_failback` was added to the `bdev_nvme_set_options` RPC to disable 280automatic failback. 281 282### idxd / dsa 283 284A new parameter `flags` was added to all low level submission and preparation 285APIs to enable the caller to pass operation flags per the DSA specification. 286 287A new flag 'SPDK_IDXD_FLAG_PERSISTENT' was added to let DSA know that 288the destination is persistent. 289 290The RPC `idxd_scan_accel_engine` has been renamed to `dsa_scan_accel_engine` 291 292The RPC `iaa_scan_accel_engine` has been added. 293 294Many HW related structs/functions with the name `idxd` have been renamed `dsa` 295to more accurately represent the HW they are associated with. 296 297Two new functions were added to the library `spdk_idxd_submit_compress` and 298`spdk_idxd_submit_decompress` 299 300### accel_fw 301 302A new parameter `flags` was added to accel API. 303The APIs include: 304`spdk_accel_submit_copy` 305`spdk_accel_submit_dualcast` 306`spdk_accel_submit_fill` 307`spdk_accel_submit_copy_crc32c` 308`spdk_accel_submit_copy_crc32cv` 309`spdk_accel_submit_compress` 310`spdk_accel_submit_decompress` 311 312A new flag `ACCEL_FLAG_PERSISTENT` was added to indicate the target memory is PMEM. 313 314The API `spdk_accel_get_capabilities` has been removed. 315 316### crypto 317 318Support for AES_XTS was added for MLX5 polled mode driver (pmd). 319 320bdev_crypto_create RPC now requires hexlified 'key' and 'key2' params for all pmd drivers. 321Unhexlifying is performed during RPC command processing and the vbdev crypto module runs on 322binary keys as before. 323 324### bdev 325 326Added a timeout option to the `bdev_get_bdevs` RPC. It allows the user to specify the amount of 327time to wait until a bdev with a given name appears in the system. 328 329### bdev_nvme 330 331Added `bdev_nvme_add_error_injection` and `bdev_nvme_remove_error_injection` RPCs to add and 332remove NVMe error injections. 333 334New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 335added to the RPC `bdev_nvme_set_options`. They can be overridden if they are given by the RPC 336`bdev_nvme_attach_controller`. 337 338### blobstore 339 340New functions `spdk_blob_io_writev_ext` and `spdk_blob_io_readv_ext` are added. The new functions accept 341`spdk_blob_ext_io_opts` structure with extended IO request options. 342 343### event 344 345Added `msg_mempool_size` parameter to `spdk_reactors_init` and `spdk_thread_lib_init_ext`. 346The size of `g_spdk_msg_mempool` can now be controlled through the same-named 347user option of `spdk_app_opts` structure. 348 349### nvme 350 351The API `spdk_nvme_ctrlr_prepare_for_reset()` was deprecated. The functionality provided by the 352`spdk_nvme_ctrlr_prepare_for_reset()` was merged into the API `spdk_nvme_ctrlr_disconnect()`. 353 354### nvmf 355 356Removed deprecated max_qpairs_per_ctrlr parameter from nvmf_create_transport RPC. Use 357max_io_qpairs_per_ctrlr instead. 358 359Deprecated the ability for hosts to connect to the discovery subsystem automatically on any 360existing listener. Users should now explicitly add listeners for the discovery subsystem. 361Host can still connect to the discovery subsystem as before, but a warning message will be 362emitted if no listener was configured for the transport ID of the incoming connection. 363 364Added adaptive interrupt feature for vfio-user transport. New parameter `disable_adaptive_irq` 365is added to the RPC `nvmf_create_transport`. 366 367### thread 368 369Added `spdk_thread_exec_msg()` API. 370 371### scheduler 372 373`framework_set_scheduler` can now be called after application initialization. 374Added callbacks to set custom parameters specific for each scheduler implementation 375and `framework_get_scheduler` to retrieve them. 376 377Added `dynamic` scheduler options: load_limit, core_limit, core_busy. Their descriptions 378are available in JSON-RPC document, in section 379[framework_set_scheduler](jsonrpc.html#rpc_framework_set_scheduler). 380 381### raid 382 383Add concat as a special raid module. The concat module could create a virtual bdev. The 384virtual bdev combines multiple underlying bdevs together. The layout of the underlying 385bdevs is one after another. The concat bdev is extendable. When the free space of the 386concat bdev is not enough, the user can deconstruct the concat bdev, then reconstruct it 387with an additional underlying bdev. 388 389### sock 390 391Allow MSG_ZEROCOPY flag to be set or not according to data size, which can be enabled and 392set by setting "zerocopy_threshold". zerocopy_threshold = 0 means disable this function; 393zerocopy_threshold > 0 means enable it and use this value as the threshold. 394 395### rpc 396 397Introduced `zerocopy_threshold` to enable zerocopy on send for server sockets according to 398data size to be flushed. 399 400## v22.01 401 402### accel 403 404The batching capability was removed. Batching is now considered an implementation 405detail of the low level drivers. 406 407### bdev 408 409The NVMe bdev module supports multipath and improved I/O error resiliency. 410 411The parameter `retry_count` of the RPC `bdev_nvme_set_options` was deprecated and will be 412removed in SPDK 22.04, and the parameter `transport_retry_count` is added and used instead. 413 414An new parameter `bdev_retry_count` is added to the RPC `bdev_nvme_set_options`. 415 416New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 417added to the RPC `bdev_nvme_attach_controller`. 418 419An new parameter `num_io_queues` is added to `bdev_nvme_attach_controller` RPC to allow specifying amount 420of requested IO queues. 421 422Added `key_file` parameter to the `rbd_register_cluster` RPC. It is an optional parameter to 423specify a keyring file to connect to a RADOS cluster. 424 425Added `spdk_bdev_get_zone_id` API to retrieve zone_id for a certain LBA. 426 427### bdev_aio 428 429Added `bdev_aio_rescan` RPC to allow rescanning the size of aio bdev. 430 431### bdev_nvme 432 433Added discovery service to bdev_nvme module, which now can connect to a discovery controller. 434bdev_nvme will connect to all current and future subsystems in the discovery controller. 435Added `bdev_nvme_start_discovery` and `bdev_nvme_stop_discovery` RPC. 436 437### dpdk 438 439Updated DPDK submodule to DPDK 21.11. 440 441### env 442 443Added `spdk_pci_for_each_device`. 444 445Removed `spdk_pci_get_first_device` and `spdk_pci_get_next_device`. These APIs were unsafe, because 446they did not account for PCI devices being inserted or removed while the caller was using handles 447returned from these APIs. Existing users of these APIs should switch to `spdk_pci_for_each_device` 448instead. 449 450Added 3 experimental APIs to handle PCI device interrupts (`spdk_pci_device_enable_interrupt`, 451`spdk_pci_device_disable_interrupt`, `spdk_pci_device_get_interrupt_efd`). 452 453Added `framework_get_pci_devices` RPC to list PCIe devices attached to an SPDK application. 454 455### idxd 456 457Many APIs are now vectored rather than scalar, meaning they take iovecs instead of individual pointers. 458 459### json 460 461Added `spdk_json_write_bytearray` API to serialize a buffer as a hex string. 462 463### nvme 464 465API `spdk_nvme_trtype_is_fabrics` was added to return existing transport type 466is fabric or not. 467 468API `spdk_nvme_poll_group_remove` was limited to be available only for a 469disconnected qpair in the group. 470 471New APIs, `spdk_nvme_ctrlr_disconnect`, `spdk_nvme_ctrlr_reconnect_async`, and 472`spdk_nvme_ctrlr_reconnect_poll_async`, have been added to improve error recovery, and 473the existing APIs,`spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async` 474were deprecated. 475 476Added `spdk_nvme_ctrlr_get_discovery_log_page` API for getting the full discovery log page 477from a discovery controller. 478 479### nvmf 480 481Added support for zero-copy operations in the NVMe-oF TCP target. It can be enabled via 482the `zcopy` parameter when creating a transport. The zero-copy operations are only used 483for requests not using in-capsule data. 484 485Added a `subsystem` parameter to `spdk_nvmf_transport_stop_listen_async`. When not NULL, 486it will only disconnect qpairs for controllers associated with the specified subsystem. 487 488Removed accept poller from transport layer. Each transport can have its own policy of 489handling new connections. To notify transport layer `spdk_nvmf_poll_group_add` and 490`spdk_nvmf_tgt_new_qpair` can be used. 491 492### scsi 493 494Structure `spdk_scsi_lun` has been extended with new member `resizing` so that SCSI layer now reports 495unit attention for disk resize. 496 497### trace 498 499Added `spdk_trace_create_tpoint_group_mask` to return tracepoint group mask from 500a tracepoint group name. 501 502Added `trace_set_tpoint_mask` and `trace_clear_tpoint_mask` RPC to allow enabling 503individual traces. 504 505### util 506 507Added `spdk_ioviter_first` and `spdk_ioviter_next` to iterate over two iovecs and 508yield pointers to common length segments. 509 510### sock 511 512A new parameter `hint` is added to `spdk_sock_get_optimal_sock_group`. It allows to suggest 513a poll group when no optimal poll group is found. 514 515## v21.10 516 517Structure `spdk_nvmf_target_opts` has been extended with new member `discovery_filter` which allows to specify 518filtering rules applied during discovery log generation. Refer to `enum spdk_nvmf_tgt_discovery_filter` for more info. 519 520### bdev 521 522New API `spdk_bdev_get_memory_domains` has been added, it allows to get SPDK memory domains used by bdev. 523 524New API functions `spdk_bdev_readv_blocks_ext` and `spdk_bdev_writev_blocks_ext` have been added. 525These functions accept `spdk_bdev_ext_io_opts` structure with extended IO request 526options, e.g. DMA memory domain which describes data that may belong to another memory domain and 527can't be accessed directly. 528 529Added `async_fini_start` to allow bdev modules to complete the `fini_start` asynchronously, 530with new `spdk_bdev_module_fini_start_done` API. 531 532Deprecated `spdk_bdev_module_finish_done()` API, which will be removed in SPDK 22.01. 533Bdev modules should use `spdk_bdev_module_fini_done()` instead. 534 535The `ocssd` bdev has been removed. The Open Channel specification has been largely superseded by 536zoned namespaces, few if any devices have been brought to market, and there is little reason to 537continue to support this. OCSSD support in the nvme driver will remain for now. 538 539### dma 540 541A new library, lib/dma, has been added. This library provides the necessary infrastructure for 542handling systems and devices with multiple memory domains. For example, a PCIe add-in card with an 543SoC may be running SPDK on the SoC. That SoC has its own local memory, but SPDK may be controlling 544devices that can also access the host system memory. This library provides infrastructure to enumerate 545the memory domains and request hardware perform DMA transfers between them. 546 547### dpdk 548 549Updated DPDK submodule to DPDK 21.08. 550 551### event 552 553Added the `disable_signal_handlers` flag to the `spdk_app_opts` struct. 554 555### idxd 556 557Added `spdk_idxd_get_socket` to query the socket that the idxd device 558is on. 559 560### json 561 562Added API to allow for writing and decoding of new types: 563 564- `spdk_json_write_named_uint16` 565- `spdk_json_write_named_uint8` 566- `spdk_json_write_uint16` 567- `spdk_json_write_uint8` 568- `spdk_json_number_to_uint8` 569- `spdk_json_decode_uint8` 570 571### log 572 573Added API `spdk_log_to_syslog_level` to return syslog level based on SPDK's 574log level. 575 576### nvme 577 578Added new functions `spdk_nvme_zns_set_zone_desc_ext` and `spdk_nvme_zns_ext_report_zones` 579to set zone descriptor extension and to get extended zone report respectively. 580 581New API `spdk_nvme_ctrlr_get_memory_domains` has been added, it allows to get SPDK memory domains used by nvme controller. 582 583New API functions `spdk_nvme_ns_cmd_readv_ext` and `spdk_nvme_ns_cmd_writev_ext` 584have been added. These functions accept `spdk_nvme_ns_cmd_ext_io_opts` structure with extended IO request 585options, e.g. DMA memory domain which describes data that may belong to another memory domain and 586can't be accessed directly. 587 588Added a new function `spdk_nvme_ctrlr_get_regs_bpinfo` to get boot partition info of a controller. 589Added new functions `spdk_nvme_ctrlr_write_boot_partition`, 590`spdk_nvme_ctrlr_read_boot_partition_start` and `spdk_nvme_ctrlr_read_boot_partition_poll` 591to write and read the boot partitions of a controller. 592 593Added `spdk_nvme_ctrlr_get_opts` to retrieve the current controller options. 594 595Added `async_mode` to `spdk_nvme_io_qpair_opts` to enable creation of submission and completion 596queues asynchronously. This mode is currently supported at PCIe layer, 597which tracks the qpair creation with state machine and returns to the user immediately. 598Default mode is set to false to create io qpairs synchronously. 599 600Added `spdk_nvme_ctrlr_get_regs_cc` to retrieve NVMe controller CC (Configuration) register. 601 602Added `spdk_nvme_ctrlr_prepare_for_reset` to inform the driver that the application is preparing 603to reset the specified NVMe controller. This function allows the driver to make decisions 604knowing that a reset is about to happen. 605 606### nvmf 607 608Added `oncs` to `struct spdk_nvmf_ctrlr_data` so that the transport layer 609can decide support RESERVATION feature or not. 610 611An `opts_size` element was added in the `spdk_nvmf_ns_opts` structure to solve the 612ABI compatibility issue between different SPDK version. An new option `anagrpid` was 613added in the `spdk_nvmf_ns_opts` structure. 614 615An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_add_ns`. 616 617An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_listener_set_ana_state`. 618 619`spdk_nvmf_subsystem_destroy` is now can be asynchronous, it accepts a callback and callback argument. 620 621A new parameter, `poll_groups_mask` was added to the `nvmf_set_config` RPC that allows specifying 622a subset of cores for the nvmf poll groups. This helps to avoid imbalances when some cores are 623busy with periodic timer tasks that run very frequently. 624 625A new parameter, `commit` was added to `nvmf_bdev_ctrlr_end_zcopy` that allows indicating whether 626the buffers should be committed. 627 628Added `listen_dump_opts` to `spdk_nvmf_transport_ops` so that the transport layer can display 629listen opts. 630 631Added `fabrics_connect_timeout_us` to `bdev_nvme_attach_controller` RPC to allow specifying 632timeout for connect operation. 633 634Added `multipath` to `bdev_nvme_attach_controller` RPC to allow specifying multipath behavior. 635 636### scheduler 637 638New API for implementing schedulers and governors. Please see `include/spdk/scheduler.h` for details. 639 640### scsi 641 642New functions, `spdk_scsi_dev_get_first_lun` and `spdk_scsi_dev_get_next_lun` 643have been added to iterate LUNs of a SCSI device. 644 645Each SCSI device supports 256 LUNs at the maximum now and the macro constant 646`SPDK_SCSI_DEV_MAX_LUN` was removed. 647 648### trace_parser 649 650A new library, lib/trace_parser, has been added. This library provides functions that parse traces 651recorded by an SPDK application. That includes merging traces from multiple cores, 652sorting them by their timestamp and constructing trace entries spanning across multiple buffers. 653 654### util 655 656The `spdk_fd_group_add` API now takes a `name` parameter. 657 658## v21.07 659 660### accel_fw 661 662Added API `spdk_accel_submit_copy_crc32c` to perform a CRC32C while copying data. 663 664Added API `spdk_accel_batch_prep_copy_crc32c` to batch CRC32C + copy commands. 665 666Added API `spdk_accel_submit_copy_crc32cv` to submit chained CRC32C + copy commands. 667 668### bdev 669 670Change `spdk_bdev_read_blocks_with_md` arg offset definition from int64_t to uint64_t. 671 672Red-black tree has been used to organize the bdev names and aliases uniformly 673to provide faster lookup. 674 675Removed ZCOPY emulation: The bdev module can be checked to see if it supports ZCOPY 676and if not supported then use existing READ/WRITE commands. 677 678Added iov to spdk_bdev_zcopy_start. 679 680### dpdk 681 682Updated DPDK submodule to DPDK 21.05. 683 684### idxd 685 686Remove the `probe_cb` parameter in `spdk_idxd_probe` function. And remove the definition 687of `spdk_idxd_probe_cb` function pointer. It should be implemented in idxd_user.c. 688 689Added API `spdk_idxd_submit_copy_crc32c` to perform a CRC32C while copying data. 690 691Added API `spdk_idxd_batch_prep_copy_crc32c` to prepare a batch operation to perform 692a CRC32C while copying data. 693 694We have userspace idxd driver under accel_engine library (module/accel/idxd/accel_engine). 695And kernel idxd driver usage is also added under the accel_engine library. This approach 696implemented a wrapper library to use IDXD device by leveraging the kernel DSA driver in 697SPDK idxd library (lib/idxd). Then users can leverage the RPC later to configure how to 698use the DSA device by user space driver or kernel driver. 699 700### init 701 702Added new `init` library that initializes the SPDK subsystems, which previously was 703internal to application framework. That functionality has been made public and for 704use by applications which don't leverage SPDK's application framework. 705 706### iscsi 707 708New parameters, `pdu_pool_size`, `immediate_data_pool_size`, and `data_out_pool_size`, 709were added to the RPC `iscsi_set_options` to run iSCSI target with varying amount of 710available memory. 711 712### json 713 714Added API `spdk_json_write_named_uint128` and `spdk_json_write_uint128` to perform 715the uint128 related data. 716 717### net 718 719Removed deprecated `net` library. 720 721Added support for ZCOPY. ZCOPY is used in preference to READ and WRITE if the 722bdev module supports ZCOPY and the bdev module has ZCOPY enabled. 723 724### nvme 725 726`spdk_nvme_map_prps` and `spdk_nvme_map_cmd` were moved to nvmf/vfio-user as internal APIs 727as vfio-user is the only user for the above two APIs. 728 729Added a new function `spdk_nvme_ns_cmd_copy` to submit a Simple Copy Command to a Namespace. 730 731Update the `spdk_nvme_generic_command_status_code` structure with new status code 732according to the definition in NVMe 1.4 spec. 733 734`spdk_nvme_ctrlr_get_default_ctrlr_opts` now sets `use_cmb_sqs` to false. This means 735that if a controller has a CMB and supports SQs in the CMB, SPDK will not use 736the CMB for SQs by default - the user must set `use_cmb_sqs` to true in 737the `spdk_nvme_ctrlr_opts` structure prior to controller attach. 738 739Add a new function `spdk_nvme_detach_poll` to simplify a common use case to continue 740polling until all detachments complete. 741 742Added new argument `timeout_admin_us` to `spdk_nvme_ctrlr_register_timeout_callback` so callers 743can specify a different timeout for admin commands vs. io commands. 744 745An existing function `spdk_nvme_detach_async` was updated to add one or more detachments 746to an active context while it is being polled. 747 748Rename a variable in the member `cmic` of the struct `spdk_nvme_ctrlr_data` from 749`multi_host` to `multi_ctrlr`. The variable means that the NVM subsystem may have two or 750more controllers if set to 1. However `multi_host` had indicated a particular use case 751such that the NVM subsystem is used by multiple hosts. 752 753A new option `disable_read_ana_log_page` was added to struct `spdk_nvme_ctrlr_opts` to disable 754reading ANA log page. The upper layer is expected to read ANA log page instead if `true`. 755The default value is `false`. 756 757New APIs, `spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async`, have been added to 758reset a controller asynchronously. 759 760New RPC `bdev_nvme_reset_controller` was added, to reset an NVMe controller. 761 762Added `spdk_nvme_ns_get_nguid` function to get NGUID for the given namespace. 763 764Added `spdk_nvme_ctrlr_is_fabrics` function to indicate whether a ctrlr handle 765is associated with a fabrics controller. 766 767### nvmf 768 769Added `min_cntlid` and `max_cntlid` to `nvmf_create_subsystem` to limit the controller ID range. 770Added `spdk_nvmf_subsystem_get_min_cntlid` and `spdk_nvmf_subsystem_get_max_cntlid` to request those values. 771 772`spdk_nvmf_request_get_buffers_multi` API is removed. 773 774Added the `nvmf_set_crdt` RPC for setting command retry delay times. 775 776Expanded `spdk_nvmf_poll_group_stat` with current qpair count statistics. 777 778Removed following deprecated APIs: 779 780- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 781- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 782- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 783- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 784- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 785- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 786- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 787 788### rpc 789 790New RPC `bdev_rbd_register_cluster` and `bdev_rbd_unregister_cluster` was added, it allows to create 791and delete the rados object cluster, then users can choose the cluster to create related rbd 792device. 793 794Revised `bdev_rbd_create` parameter, it allows to use an optional parameter `--cluster-name` 795to create a rbd bdev with an already registered Rados Cluster Object. 796 797New RPC `bdev_rbd_get_clusters_info` was added, it allows to get the info of the registered 798Rados Cluster names. 799 800New optional parameter, `timeout_admin_us`, added to the bdev_nvme_set_options RPC. 801 802Revised a parameter `--stripe-size_kb` to `--stripe-size-kb` of `bdev_raid_create` method 803provided in `scripts/rpc.py` for consistency. 804 805An new optional parameter `config_kernel_mode` was added to the RPC `idxd_scan_accel_engine`, 806and this is used to enable using the kernel mode IDXD driver. 807 808Removed deprecated parameter `enable-zerocopy-send` of RPC `sock_impl_set_options`, 809use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 810Removed deprecated parameter `disable-zerocopy-send` of RPC `sock_impl_set_options`, 811use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 812 813### rpm 814 815Removed deprecated `pkg/spdk.spec`, please use `rpmbuild/spdk.spec` instead. 816See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 817 818### thread 819 820Red-black tree has been used for timed pollers to provide faster insertion and deletion 821and for io_devices to provide faster lookup. 822 823Removed `spdk_io_channel` structure from public header and moved it to thread_internal.h. 824 825### trace 826 827Added `spdk_trace_register_description_ext` function to register variable number of tracepoint arguments. 828 829Added ability to chain multiple trace entries together to extend the size of the argument buffer. 830 831### util 832 833Red-black tree macros has been added by using the macros provided by the FreeBSD operating system 834under the same BSD license. 835 836Add an new macro `SPDK_SIZEOF_MEMBER` to get the size of a member of a struct. 837 838`spdk_crc32c_iov_update` function was added to support calculating the crc32c of the iovs. 839 840Added zipf random number generator with power law probability distribution. 841When applied to performance testing of block devices, it will select blocks over 842the full range of LBAs, but will more frequently select lower-numbered LBAs. 843 844## v21.04 845 846### accel 847 848Two new accelerated crc32 functions `spdk_accel_submit_crc32cv` and 849`spdk_accel_batch_prep_crc32cv` are added in order to provide the 850chained accelerated CRC32 computation support. 851 852### bdev 853 854For `bdev_ocssd_create` RPC, the optional parameter `range` was removed. 855Only one OCSSD bdev can be created for one OCSSD namespace. 856 857Removed the `spdk_bdev_open` from bdev library API. 858Removed the `spdk_vbdev_register` and `spdk_bdev_part_base_construct` from bdev module API. 859Removed the `config_text` function for bdev modules to report legacy config. 860 861Added `spdk_bdev_get_max_active_zones` API to display maximum number active zones of a given bdev. 862 863Added `spdk_bdev_get_max_zone_append_size` API to display maximum zone append data transfer size. 864 865### bdev_nvme 866 867Added support for zoned namespaces. 868 869### blobstore 870 871Removed the `spdk_bdev_create_bs_dev_from_desc` and `spdk_bdev_create_bs_dev` API. 872 873### env 874 875Added `spdk_pci_device_allow` API to allow applications to add PCI addresses to 876the allowed list after the application has started. 877 878Removed the `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts`. 879 880Added hotplug support based on uevent in `pci_event.c`. Added usage of this functionality in 881nvme, virtio-scsi and virtio_blk libraries. Please see the new API `spdk_pci_event_listen`, 882`spdk_pci_get_event`, `spdk_pci_register_error_handler` and `spdk_pci_unregister_error_handler`. 883 884### event 885 886Removed the `config_file`, `max_delay_us`, `pci_whitelist` 887and `pci_blacklist` members of struct `spdk_app_opts`. 888 889### idxd 890 891A new API `spdk_idxd_device_needs_rebalance` was added so that users of the library 892can know whether they need to rebalance the flow control for the channel 893that was just added/removed. This is based on how the low level library 894shares devices amongst channels. 895 896The API `spdk_idxd_reconfigure_chan` had the `num_channels` removed as this 897is now tracked in the library. The app makes use the new API above to 898determine whether to rebalance or not. This applies to `spdk_idxd_configure_chan` 899as well. 900 901The API `spdk_idxd_put_channel` now returns the rebalance state for the 902underlying device. 903 904### iscsi 905 906A security vulnerability has been identified and fixed in the SPDK iSCSI target. 907A TEXT PDU with no data, but CONTINUE flag set, would result in a NULL pointer dereference 908and crash the SPDK iSCSI target process. All users of the SPDK iSCSI target 909are recommended to update. All SPDK versions <= v21.01 are affected. 910 911### net 912 913The net library is deprecated and will be removed in the 21.07 release. 914 915### nvme 916 917Added a new function `spdk_nvme_ctrlr_get_regs_pmrcap` to get the PMR capabilities. 918 919Added an accelerated table pointer in `spdk_nvme_poll_group` 920which can be used to provide the accelerated functions by users with 921hardware engine, such as crc32c accelerated function. 922 923Added new functions `spdk_nvme_ctrlr_get_pmrsz`, `spdk_nvme_ctrlr_enable_pmr`, 924`spdk_nvme_ctrlr_disable_pmr`, `spdk_nvme_ctrlr_map_pmr` and `spdk_nvme_ctrlr_unmap_pmr`. 925 926Added NVMe transport operations to enable, disable, map and unmap the PMR. 927 928Added `spdk_nvme_qpair_get_optimal_poll_group` function and `qpair_get_optimal_poll_group` 929function pointer in spdk_nvmf_transport_ops structure in order to add the qpair to the most 930suitable polling group. 931 932Added OPTPERF and namespace optimal performance fields to nvme_spec.h. 933 934Added `spdk_nvme_set_hotplug_filter` API to allow applications to choose which 935hot-inserted SSDs should be probed. This is useful for use cases where multiple 936independent SPDK processes are running on one node. The filter function can 937then be implemented in these processes to decide which SSDs to probe based on 938the new SSD's PCI address. 939 940New functions `spdk_nvme_poll_group_get_stats` and `spdk_nvme_poll_group_free_stats` 941were added. These functions allow to get transport statistics per NVME poll group. 942 943Added `spdk_nvme_map_cmd` API to map the NVMe command with SGL cases. 944 945Added support for vector variant of ZNS zone append commands with new API 946`spdk_nvme_zns_zone_appendv` and `spdk_nvme_zns_zone_appendv_with_md`. 947 948Added `spdk_nvme_zns_ns_get_max_open_zones` and `spdk_nvme_zns_ns_get_max_active_zones` API, 949to display maximum number of open and active zones of the given namespace. 950 951Added `spdk_nvme_zns_ns_get_zone_size_sectors` API to provide size of zone in number of 952sectors. 953 954Added `spdk_nvme_qpair_get_id` API to display the ID of the specified qpair. 955 956### nvmf 957 958Removed the `spdk_nvmf_tgt_listen` and `spdk_nvmf_subsystem_add_ns` API. 959 960Added new APIs: 961 962- `spdk_nvmf_poll_group_dump_stat` (function in `nvmf.h`). 963- `poll_group_dump_stat` (transport op in `nvmf_transport.h`). 964 965The following APIs have been deprecated and will be removed in SPDK 21.07: 966 967- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 968- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 969- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 970- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 971- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 972- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 973- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 974 975See header files for details. 976 977The `trtype` field in JSON returned by `nvmf_get_stats` RPC contains now the name of the transport, 978which is the same as the type for defined transports and more informative for a custom transport. 979 980Added `hdgst` and `ddgst` parameters to `bdev_nvme_attach_controller` RPC in order change 981state of TCP header and data digest. 982 983Added `num_cqe` parameter to `nvmf_create_transport` RPC to set number of completion queues (CQ) 984for RDMA transport. Useful when CQ resize operation is not supported, for example iWARP. 985 986### ocf 987 988Updated OCF submodule to v20.12.2 989 990Added `bdev_ocf_set_cache_mode` RPC to dynamically switch cache mode of OCF bdev. 991 992### opal 993 994Removed the `spdk_opal_supported` API. 995 996### raid 997 998For `bdev_raid_create` RPC, the deprecated parameter `strip_size` was removed. 999 1000### rpc 1001 1002New RPC `bdev_nvme_get_transport_statistics` was added, it allows to get transport statistics 1003of nvme poll groups. 1004 1005Parameter `enable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1006use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 1007Parameter `disable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1008use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 1009 1010Added cmd_parser.py used to parse the args from argparse. There are 1011two benefit to use command parser: 1012 1013- Simplify the definition of rpc method. It will reduce the rpc method code. 1014- Make the rpc call more versatile. User can add private args into rpc method. 1015 1016### rpm 1017 1018Added support for new RPM spec, rpmbuild/spdk.spec, which can be used for packaging the 1019SPDK. The pkg/spdk.spec is considered to be deprecated and scheduled for removal in SPDK 21.07. 1020See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 1021 1022### sock 1023 1024The type of `enable_placement_id` in struct `spdk_sock_impl_opts` is changed from 1025bool to int. We can use RPC to configure different value of `enable_placement_id`. 1026Then we can leverage SO_INCOMING_CPU to get placement_id, which aims to utilize 1027CPU cache locality, enabled by setting enable_placement_id=2. 1028 1029A new socket placement mode called PLACEMENT_MARK has been added. Some NICs allow 1030sockets to be marked using the SO_MARK socket option as a hint for which hardware 1031queue they should be associated with. This mode leverages that by setting the same 1032value for all sockets within a poll group. 1033 1034New parameters `enable_zerocopy_send_server` and `enable_zerocopy_send_client` were added 1035to struct spdk_sock_impl_opts, these parameters enable or disable zero copy send for server 1036and client sockets which are created using `spdk_sock_listen` and `spdk_sock_listen_ext` (server); 1037`spdk_sock_connect` and `spdk_sock_connect_ext` (client) functions. Existing parameter 1038`enable_zerocopy_send` enables or disables zero copy send for both server and client sockets. 1039 1040### thread 1041 1042A new API `spdk_io_channel_get_io_device` was added to get the io_device for the specified 1043I/O channel. 1044 1045Added `spdk_thread_set_interrupt_mode` function in order to set present spdk_thread into 1046interrupt mode or back to poll mode. It is valid only when thread interrupt facility is 1047enabled by `spdk_interrupt_mode_enable`. 1048 1049Added `spdk_poller_register_interrupt` function to mark that the poller is capable of 1050entering interrupt mode. Callback function will be called when the poller must transition 1051into or out of interrupt mode. 1052 1053### virtio 1054 1055Added the `bdev_virtio_blk_set_hotplug` RPC for the virtio blk pci device. 1056 1057## v21.01 1058 1059### bdev 1060 1061An `opts_size` element was added in the `spdk_bdev_opts` structure to solve the 1062ABI compatibility issue between different SPDK version. And also add `opts_size` 1063parameter in spdk_bdev_get_opts function. Two fields `small_buf_pool_size` and 1064`large_buf_pool_size` were added into spdk_bdev_opts, which were used to determine 1065the small and large buffer pool size of the whole bdev module. 1066 1067A new API `spdk_bdev_wait_for_examine` was added to allow for checking state of 1068examine process. Along with corresponding `bdev_wait_for_examine` RPC, which 1069is now always called during `spdk_bdev_subsystem_config_json` making sure 1070every bdev is ready to be used. 1071 1072A new API `spdk_bdev_io_get_aio_status` was added for getting the status of 1073bdev_io as Linux AIO errno. Also `spdk_bdev_io_complete_aio_status` function 1074and `SPDK_BDEV_IO_STATUS_AIO_ERROR` were added for bdev module to complete 1075a bdev_io with Linux AIO errno. 1076 1077A new API `spdk_bdev_get_module_name` was added for getting block device module name. 1078Also `spdk_bdev_get_module_ctx` was added to obtain a bdev module context for 1079the block device opened by the specified descriptor. 1080 1081Added `max_segment_size` and `max_num_segments` to `spdk_bdev` structure for 1082bdev modules to specify splitting requirements. 1083 1084### blob 1085 1086An `opts_size` element was added in the `spdk_bs_opts` structure to solve the 1087ABI compatibility issue between different SPDK version. And also add `opts_size` 1088parameter in `spdk_bs_opts_init` function. 1089 1090An `opts_size` element was added in the `spdk_blob_opts` structure to solve the 1091ABI compatibility issue between different SPDK version. And also add `opts_size` 1092parameter in `spdk_blob_opts_init` function. 1093 1094An `opts_size` element was added in the `spdk_blob_open_opts` structure to solve the 1095ABI compatibility issue between different SPDK version. And also add `opts_size` 1096parameter in `spdk_blob_open_opts_init` function. 1097 1098### build 1099 1100SPDK now generates pkg-config files to simplify the process of determining which 1101libraries must be linked into an SPDK application. 1102 1103### dpdk 1104 1105Updated DPDK submodule to DPDK 20.11. 1106 1107Removed `--with-igb-uio-driver` configure option. Since DPDK 20.11 igb_uio driver 1108was moved to separate dpdk-kmods repository. If required, users need to refer to 1109dpdk-kmods repository to build the driver. 1110 1111### env 1112 1113The `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts` 1114have been deprecated. The new members are named `pci_allowed`, `pci_blocked` and 1115`main_core` respectively. 1116 1117An `opts_size` element was added in the `spdk_app_opts` structure 1118to solve the ABI compatibility issue between different SPDK version. An `opts_size` 1119parameter is added into `spdk_app_opts_init` function. 1120 1121### event 1122 1123The `pci_whitelist` and `pci_blacklist` members of struct `spdk_app_opts` have been 1124deprecated. The new members are named `pci_allowed` and `pci_blocked` respectively. 1125 1126The `--pci-blacklist` command line option has been deprecated, replaced with 1127`--pci-blocked`. 1128 1129The `--pci-whitelist/-W` command line options have been deprecated, replaced with 1130`--pci-allowed/-A`. 1131 1132Added new experimental `dynamic` scheduler that rebalances idle threads, adjusts CPU frequency 1133using dpdk_governor and turns idle reactor cores to interrupt mode. Please see 1134[scheduler documentation](https://www.spdk.io/doc/scheduler.html) for details. 1135 1136## ioat 1137 1138The PCI BDF whitelist option has been removed from the `ioat_scan_accel_engine` RPC. 1139IOAT PCI functions can still be allowed or denied using SPDK application command 1140line options. 1141 1142### isa-l 1143 1144Updated ISA-L submodule to v2.30.0. 1145 1146### json 1147 1148A new API `spdk_jsonrpc_send_bool_response` was added to allow sending response for 1149writing json bool results into one function. 1150Update API `bdev_nvme_set_options` and add a `keep_alive_timeout_ms` parameter. Now you 1151can specify the `keep_alive_timeout` before creating NVMe bdev. 1152 1153### nbd 1154 1155Change the return type of function `spdk_nbd_stop` from void to int. And update the 1156`spdk_nbd_fini` with two parameters to make its behavior from sync to async. 1157 1158### nvme 1159 1160Directives support was added to the NVMe driver. 1161 1162Two async APIs `spdk_nvme_ctrlr_cmd_directive_receive` and `spdk_nvme_ctrlr_cmd_directive_send` 1163are added for Directive Send and Directive Receive command, respectively. 1164 1165Added a new function `spdk_nvme_ctrlr_reset_subsystem` to perform a NVMe 1166subsystem reset. Note: The NVMe-oF target does not support the subsystem reset yet. 1167 1168Add a new function `spdk_nvme_bytes_to_numd` to transfer bytes to number of 1169dwords. 1170 1171Added a new custom transport `SPDK_NVME_TRANSPORT_VFIOUSER` to enable NVMe 1172driver running with NVMe over vfio-user target. 1173 1174Added the vfio-user custom transport implementation in NVMe driver which can connect 1175to NVMe over vfio-user target via vfio-user transport. 1176 1177Added a new function `spdk_nvme_ctrlr_set_remove_cb` to remove or override 1178`remove_cb` and `remove_ctx` specified when the controller was probed. 1179 1180Added support for ZNS zone append command with new API `spdk_nvme_zns_zone_append` and 1181`spdk_nvme_zns_zone_append_with_md`. 1182 1183### nvmf 1184 1185`nvmf_fc_lld_fini` now takes callback and hence updating FC Broadcom LLD driver 1186to the latest is required. 1187 1188FC transport supported primary tagging and VMID. 1189 1190Broadcom FC LLD driver and SPDK NVMe-oF FC transport consolidated one LLD API, 1191`nvmf_fc_init_poller_queues` into another LLD API `nvmf_fc_init_q`. 1192Hence updating Broadcom FC LLD driver to the latest is required. 1193 1194The functions `destroy` and `qpair_fini` in the transport interface now accept a 1195`cb_fn` and `cb_arg` to call upon completion, and their execution can be asynchronous. 1196 1197The SPDK NVMe-oF target now supports async event notification for discovery log changes. 1198This allows the initiator to create persistent connection to discovery controller and 1199be notified of any discovery log changes. 1200 1201An `opts_size` element was added in the `spdk_nvmf_transport_opts` structure 1202to solve the ABI compatibility issue between different SPDK version. And also add 1203`opts_size` parameter in `spdk_nvmf_transport_opts_init` function. 1204 1205Added a new custom vfio-user transport implementation in NVMe-oF which can provide 1206emulated NVMe devices to QEMU and SPDK NVMe driver. 1207 1208Added new API `spdk_nvmf_tgt_listen_ext` that uses `spdk_nvmf_listen_opts` to allow 1209passing generic options to specific transport. This functionality replaces one provided 1210with `nvmf_subsystem_set_options`. Also removed `nvmf_subsystem_set_options` RPC 1211that allowed transport specific options within subsystem. 1212 1213The `trsvcid` in `nvmf_subsystem_add_listener` RPC is now optional. 1214 1215Pausing a subsystem now only pauses all admin queues. This allows for the 1216management of hosts and listeners, plus the addition of namespaces without a 1217full subsystem pause. Additionally, the target now allows for pausing 1218individual namespaces within a subsystem. To remove a namespace from a 1219subsystem, only the specific namespace must be paused. I/O will continue to 1220other namespaces while these operations execute. 1221 1222### rpc 1223 1224An new optional parameter `wait` was added to the RPC `iscsi_create_portal_group`, 1225and an new RPC `iscsi_start_portal_group` was added. They will be used not to start 1226listening on portals for a portal group until all associated target nodes are created 1227at startup, otherwise some iSCSI initiators may fail to re-login when SPDK iSCSI 1228target application restarts. 1229 1230Two optional parameter `--small-buf-pool-size` and `--large-buf-pool-size` were added 1231into `bdev_set_options` function. 1232 1233### vhost 1234 1235Added optional `packed_ring_recovery` parameter to `vhost_create_blk_controller` RPC 1236enabling packed ring live recovery. 1237 1238## v20.10 1239 1240### accel 1241 1242New APIs `spdk_accel_batch_cancel` and `spdk_idxd_batch_cancel` were added to allow 1243cancelling a batch sequence. 1244 1245### bdev 1246 1247A new `spdk_bdev_part_base_construct_ext` function has been added and the 1248`spdk_bdev_part_base_construct` has been deprecated. The 1249`spdk_bdev_part_base_construct_ext` function takes bdev name as an argument instead 1250of bdev structure to avoid a race condition that can happen when the bdev is being 1251removed between a call to get its structure based on a name and actually opening it. 1252 1253Removed `spdk_bdev_config_text` function for bdev modules to report legacy config. 1254 1255### blobstore 1256 1257A new `spdk_bdev_create_bs_dev_ext` function has been added and `spdk_bdev_create_bs_dev_from_desc` 1258function has been deprecated. 1259 1260### dpdk 1261 1262Updated DPDK submodule to DPDK 20.08. 1263 1264### env 1265 1266A new API `spdk_mem_get_fd_and_offset` was added to get file descriptor and offset for 1267a given virtual address. 1268 1269### event 1270 1271Removed `spdk_subsystem_config` callback for submodules as part of legacy config removal. 1272 1273Removed `spdk_app_get_running_config` function that printed configuration in legacy format, 1274and removed `usr1_handler` from `struct spdk_app_opts` callback that was used to call it. 1275 1276Added SPDK thread scheduler framework which is used to rebalance load among CPU cores. 1277The scheduler implementation is pluggable and two schedulers are added first, `static` scheduler 1278and `gscheduler` scheduler. 1279 1280`static` scheduler is used by default and does not reschedule threads and keeps the initial 1281assignments. 1282 1283CPU frequency governor framework and implementation are also added. 1284`dpdk_governor` is the first implementation and is based on rte_power library of DPDK. 1285`gscheduler` scheduler utilizes the CPU frequency governor. 1286 1287Scheduler and governor frameworks are experimental features. 1288 1289### fio 1290 1291Bdev fio_plugin no longer supports legacy configuration files. Options `spdk_conf` and 1292`spdk_json_conf` have to point to valid JSON configuration. 1293 1294### gpt 1295 1296Removed option to disable GPT probing via legacy configuration. Please use generic 1297bdev functionality to replicate the behavior. See `bdev_set_options --disable-auto-examine` 1298and `bdev_examine` RPC. 1299 1300### intel-ipsec-mb 1301 1302Updated intel-ipsec-mb submodule to v0.54 1303 1304### isa-l 1305 1306Updated ISA-L submodule to v2.29.0. 1307 1308### iscsi 1309 1310The iSCSI target supports iSCSI login redirection feature to achieve fault tolerance 1311and load balancing particularly in scale out iSCSI target system, which runs multiple 1312SPDK iSCSI targets. 1313 1314### json 1315 1316A new API `spdk_json_decode_object_relaxed` was added to allow parsing json values 1317even if decoder for a given key is not found. 1318 1319A new API `spdk_json_free_object` was added to free memory allocated by `spdk_json_decode_object`. 1320 1321### log 1322 1323A log flag structure, `spdk_log_register_flag`, `spdk_log_get_first_flag`, 1324`spdk_log_get_next_flag` and macro `SPDK_LOG_REGISTER_COMPONENT`, have been added which 1325enable registering and interacting with log flags. Added `SPDK_INFOLOG`, `SPDK_DEBUGLOG` 1326and `SPDK_LOGDUMP` macros to display logs for specified log flag. 1327 1328The log flags in the event framework `-L` and several SPDK applications has been enabled on 1329release builds of SPDK. On debug builds this option will additionally set 1330log print level to `SPDK_LOG_DEBUG`. 1331 1332A new API `spdk_log_enable_timestamps` and new RPC `log_enable_timestamps` was added to 1333toggle log timestamps. 1334 1335### miscellaneous 1336 1337The contents of the log_rpc library have been moved to the event library. The log_rpc 1338library now no longer exists. 1339 1340The contents of the app_rpc library have been moved to the event library. The app_rpc 1341library now no longer exists. 1342 1343The contents of the bdev_rpc library have been moved to the bdev library. The app_rpc 1344library now no longer exists. 1345 1346The bdevperf application now disables the zcopy API by default. Prior to this change, 1347bdevperf enabled using the zcopy API by default which caused a performance impact of 1348up to 25% on bdevs that don't natively support zcopy because the API emulates zero-copy 1349by allocating a buffer. The bdevperf `-x` param was renamed to `-Z` and the default 1350value changed to false. For bdevs that support zcopy, use the -Z flag to enable 1351using zcopy API. 1352 1353### net 1354 1355Net framework intended for use by user-space TCP stacks was removed. 1356 1357### nvme 1358 1359Zoned Namespace Command Set support has been added to NVMe driver, see `nvme_zns.h` for details. 1360 1361New APIs, `spdk_nvme_detach_async` and `spdk_nvme_detach_poll_async`, have been added to 1362detach multiple controllers in parallel to mitigate lengthy shutdown notification of 1363a few NVMe SSDs. 1364 1365New APIs, `spdk_nvme_ns_get_ana_group_id` and `spdk_nvme_ns_get_ana_state`, have been 1366added to get ANA group ID and ANA state of the given namespace, respectively. 1367 1368If no specific command set is requested (by setting the command_set member in the 1369`spdk_nvme_ctrlr_opts` structure), SPDK will automatically select the most appropriate 1370command set based on what the controller supports. 1371 1372### nvmf 1373 1374A new API `spdk_nvmf_subsystem_add_ns_ext` has been added and the API `spdk_nvmf_subsystem_add_ns` 1375has been deprecated. 1376 1377The NVMe-oF target now supports Asymmetric Namespace Access (ANA) Reporting to provide 1378multipath to NVMe-oF initiator. 1379 1380Add `no_wr_batching` parameter to `spdk_nvmf_transport_opts` struct to disable 1381Work Requests batching in RDMA transport. 1382 1383NVMf Target transports can now parse any additional JSON params in the nvmf_create_transport RPC 1384through the JSON context provided via spdk_nvmf_target_opts->transport_specific. 1385 1386A new RPC `nvmf_subsystem_set_options` was added to allow transport specific options within subsystem. 1387 1388A new API `spdk_nvmf_transport_stop_listen_async` was added to stop accepting new connections 1389at the provided address. `nvmf_subsystem_remove_listener` RPC now uses this function. 1390 1391A new API `spdk_nvmf_subsystem_disconnect_host` was added to disconnect all connections 1392originating from the provided hostnqn. `nvmf_subsystem_remove_host` RPC now uses this function. 1393 1394### ocf 1395 1396Updated OCF submodule to v20.03.1 1397 1398### rpc 1399 1400New optional parameters, `enable_placement_id` and `enable_quickack` were added to the 1401`sock_impl_set_options` RPC. 1402 1403A new RPC `bdev_examine_bdev` was added to allow users to examine a bdev explicitly. 1404It can be used only if bdev_auto_examine is set to false by the RPC `bdev_set_options`. 1405 1406New optional parameters `no_wr_batching` and `control_msg_num` were added to the RPC 1407'nvmf_create_transport'. 1408 1409New RPCs, `iscsi_target_node_set_redirect` and `iscsi_target_node_request_logout`, have 1410been added, and a new optional parameter `private` has been added to the RPC 1411`iscsi_create_portal_group` to support iSCSI login redirection feature. 1412 1413A new optional parameter `ana_reporting` was added to the RPC `nvmf_create_subsystem`, and 1414a new RPC `nvmf_subsystem_listener_set_ana_state` was added for ANA reporting. 1415 1416New RPCs, `nvmf_subsystem_get_listeners` and `nvmf_subsystem_get_qpairs`, were added to 1417retrieve configuration of the NVMe-oF subsystem. 1418 1419### scsi 1420 1421Two new APIs have been added `spdk_scsi_dev_construct_ext` and 1422`spdk_scsi_dev_add_lun_ext` that allow the upper layer(e.g. vhost-scsi) to 1423receive the notification when the scsi bdev has been resized. 1424 1425The `spdk_scsi_dev_construct` and `spdk_scsi_dev_add_lun` eventually may be 1426deprecated and removed. 1427 1428### sock 1429 1430The `enable_placement_id` field was added in the `struct spdk_sock_impl_opts` to 1431make the placement_id feature configurable by users. The default setting is 1432not enabled. 1433 1434The `enable_quick_ack` field was added in the `struct spdk_sock_impl_opts` to enable 1435or disable quick ack for the POSIX sock module. The default setting is not enabled. 1436 1437### thread 1438 1439`fd_group` is applied to support interrupt mode. 1440New APIs were added to support an experimental interrupt mode. This allows modules or 1441libraries to selectively register file descriptors that the spdk_thread can wait on, 1442as an alternative to polling. In v20.10, this functionality is enabled in a very small 1443subset of SPDK libraries and modules. 1444 1445### util 1446 1447A new utility named `fd_group` was add. It is now 1448implemented by epoll on Linux platform. It can be used by 1449spdk_thread and reactor to implement interrupt mode. 1450 1451Added `spdk_bit_pool` utility, see `bit_pool.h` for details. 1452 1453### vhost 1454 1455SPDK has switched to DPDK's rte_vhost library since 19.07 release, removed the internal 1456rte_vhost library which is used for DPDK older than 19.05, removed the experimental vhost 1457nvme target which depends on the internal rte_vhost library. 1458 1459### vpp 1460 1461Removed socket abstraction for VPP. 1462 1463## v20.07: SPDK CSI driver, new accel_fw commands, I/O abort support 1464 1465spdk_json_free_object() was added to free memory allocated by spdk_json_decode_object(). 1466 1467### accel 1468 1469A new API was added `spdk_accel_get_capabilities` that allows applications to 1470query the capabilities of the currently enabled accel engine back-end. 1471 1472A new capability, CRC-32C, was added via `spdk_accel_submit_crc32c`. 1473 1474The software accel engine implementation has added support for CRC-32C. 1475 1476A new capability, compare, was added via `spdk_accel_submit_compare`. 1477 1478The software accel engine implementation has added support for compare. 1479 1480Several APIs were added to `accel_engine.h` to support batched submission 1481of operations. 1482 1483Several APIs were added to `accel_engine.h` to support dualcast operations. 1484 1485### accel_fw 1486 1487The accel_fw was updated to no longer require the app to allocate an 1488accel_task on its behalf. All public APIs now take a callback arg as 1489the parameter that used to be the accel_task. 1490 1491The accel_fw API `spdk_accel_task_size` was removed as it is no longer 1492required. 1493 1494The accel_fw was updated to support compare, dualcast, crc32c. 1495 1496The accel_fw introduced batching support for all commands in all plug-ins. 1497See docs for detailed information. 1498 1499### bdev 1500 1501A new API `spdk_bdev_abort` has been added to submit abort requests to abort all I/Os 1502whose callback context match to the bdev on the given channel. 1503 1504### build 1505 1506The fio plugins now compile to `build/fio` and are named `spdk_bdev` and `spdk_nvme`. 1507Existing fio configuration files will need to be updated. 1508 1509### dpdk 1510 1511Updated DPDK submodule to DPDK 20.05. 1512 1513### env 1514 1515Several new APIs have been added to provide greater flexibility in registering and 1516accessing polled mode PCI drivers. See `env.h` for more details. 1517 1518### idxd 1519 1520The idxd library and plug-in module for the accel_fw were updated to support 1521all accel_fw commands as well as batching. Batching is supported both 1522through the library and the plug-in module. 1523 1524IDXD engine support for CRC-32C has been added. 1525 1526### ioat 1527 1528A new API `spdk_ioat_get_max_descriptors` was added. 1529 1530### nvme 1531 1532An `opts_size`element was added in the `spdk_nvme_ctrlr_opts` structure 1533to solve the ABI compatibility issue between different SPDK version. 1534 1535A new API `spdk_nvme_ctrlr_cmd_abort_ext` has been added to abort previously submitted 1536commands whose callback argument match. 1537 1538Convenience functions, `spdk_nvme_print_command` and `spdk_nvme-print_completion` were added 1539to the public API. 1540 1541A new function, `spdk_nvmf_cuse_update_namespaces`, updates the cuse representation of an NVMe 1542controller. 1543 1544A new function `qpair_iterate_requests` has been added to the nvme transport interface. ALl 1545implementations of the transport interface will have to implement that function. 1546 1547### nvmf 1548 1549The NVMe-oF target no longer supports connecting scheduling configuration and instead 1550always uses what was previously called "transport" scheduling. 1551 1552`spdk_nvmf_tgt_accept` no longer exists. The accept process now occurs automatically after 1553the creation of an nvmf target and queue pairs are assigned to poll groups by the underlying 1554transport. Further, `spdk_nvmf_transport_ops` has changed such that the accept function 1555pointer no longer takes a function pointer as an argument. Instead, transports should call 1556`spdk_nvmf_tgt_new_qpair` whenever they previously would have called that callback. 1557 1558The NVMe-oF target now supports aborting any submitted NVM or Admin command. Previously, 1559the NVMe-oF target could abort only Asynchronous Event Request commands. 1560 1561### rdma 1562 1563A new `rdma` library has been added. It is an abstraction layer over different RDMA providers. 1564Two providers are available - verbs (used by default when RDMA is enabled or enabled explicitly 1565using --with-rdma=verbs) and mlx5 Direct Verbs aka DV (enabled by --with-rdma=mlx5_dv). 1566Using mlx5_dv requires libmlx5 installed on the system. 1567 1568### rpc 1569 1570Parameter `-p` or `--max-qpairs-per-ctrlr` of `nvmf_create_transport` RPC command accepted by the 1571rpc.py script is deprecated, new parameter `-m` or `--max-io-qpairs-per-ctrlr` was added. 1572 1573Added `sock_impl_get_options` and `sock_impl_set_options` RPC methods. 1574 1575Command line parameters `-r` and `--rpc-socket` will longer accept TCP ports. RPC server 1576must now be started on a Unix domain socket. Exposing RPC on the network, as well as providing 1577proper authentication (if needed) is now a responsibility of the user. 1578 1579The `bdev_set_options` RPC has a new option, `bdev_auto_examine` to control the auto examine function 1580of bdev modules. 1581 1582New RPCs `sock_impl_get_options` and `sock_impl_set_options` been added to expose new socket features. 1583See `sock` section for more details. 1584 1585### sock 1586 1587Added `spdk_sock_impl_get_opts` and `spdk_sock_impl_set_opts` functions to set/get socket layer configuration 1588options. Options can be set independently for each implementation. 1589 1590Added `recv_buf_size` and 'send_buf_size' socket layer options. They are used only in posix implementation. 1591 1592Added `uring` based socket implementation, the code is located in module/sock/uring. This feature is only 1593available in Linux which requires kernel version is greater than 5.4.3. Currently, our CI pool added the uring 1594based socket tests for iSCSI target and also the tests for SPDK NVMe-oF tcp transport. 1595 1596Added `enable_recv_pipe` socket layer option to allow disabling of double buffering on receive. 1597New option is used only in posix implementation. 1598 1599Added `enable_zerocopy_send` socket layer option to allow disabling of zero copy flow on send. 1600New option is used only in posix implementation. 1601 1602### util 1603 1604Some previously exposed CRC32 functions have been removed from the public API - 1605`spdk_crc32_update`, `spdk_crc32_table_init`, and the `spdk_crc32_table` struct. 1606 1607### vhost 1608 1609The function `spdk_vhost_blk_get_dev` has been removed. 1610 1611## v20.04: SPDK Top, IDXD, NVMe qpair groups 1612 1613IDXD engine support for compare has been added. 1614 1615### configuration 1616 1617Legacy INI style configuration for SPDK applications has been deprecated and will be 1618removed in future release. 1619Please switch to JSON-RPC configuration files and/or RPC driven run-time configuration. 1620 1621### copy 1622 1623The copy engine library, modules and public APIs have been renamed. Use of the word `copy` 1624has been replaced with the word `accel` short for accelerator in preparation for adding new 1625capabilities in the future. Additionally, APIs for what was previously called the `memcpy` 1626engine have been renamed to identify the engine as a software accelerator. 1627 1628### crypto 1629 1630Support for AES_XTS was added for the QAT polled mode driver (pmd). The create RPC 1631`bdev_crypto_create` has 2 new optional parameters: cipher and key2. Cipher can be either 1632AES_CBC (default) or AES_XTS. AES_XTS is only valid when using the QAT polled mode driver. 1633The key2 parameter is the second key required for AES_XTS. 1634 1635### event 1636 1637Reactors now accumulate CPU stats and they are retrieved by the RPC `framework_get_reactors`. 1638 1639### idxd 1640 1641IDXD support was added in the form of a low level library that can directly 1642utilize the IDXD offload engine as well as an accel plug-in module allowing for use 1643with the generic accel framework API. IDXD is the first in a family of offload 1644engines that share the same interface, specifically DSA is added here. More info 1645can be found here: https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator 1646 1647Much of the implementation models IOAT, however the low level interfaces are very 1648different. The RPC to enable IDXD requires a configuration number as well. The 1649code includes two pre-defined configurations of IDXD groups/work queues/engines. A future 1650version will provide an interface to allow for setting of individual configuration 1651parameters. 1652 1653IDXD is not yet available so this feature should be considered experimental. It will 1654be built up with additional documentation as an ongoing activity. 1655 1656### iscsi 1657 1658The iSCSI target now creates a lightweight thread per poll group instead of assuming a pool 1659of lightweight threads already exist at start up time. A poll group is a collection of 1660unrelated iSCSI connections. Each poll group is only accessed from the associated 1661lightweight thread. 1662 1663### ftl 1664 1665Several changes have been made to the `spdk_ftl_conf`, `spdk_ftl_dev_init_ops`, and 1666`spdk_ftl_attrs` structs. Please see `include/spdk/ftl.h` for more details. 1667 1668### miscellaneous 1669 1670The `--json-ignore-init-errors` command line parameter has been added to ignore 1671initialization errors on JSON config load. 1672 1673The public header file io_channel.h has been removed. Please use thread.h which has the 1674exact same API. 1675 1676### nvme 1677 1678Exported internal nvme_ctrlr_cmd_security_receive/send() APIs as public APIs with "the spdk_" 1679prefix. 1680 1681Added `priority` field in `spdk_nvme_transport_id`, this field is used to specify the priority 1682of the NVMe-oF connection, and currently it is used for NVMe-oF tcp connection. 1683 1684A new poll group API has been added to allow for pooling of nvme qpairs across a single 1685entity which can be polled for completions. This new API consists of the `spdk_nvme_poll_group` 1686family of functions. As a result of this new API, all NVMe transports are expected to implement 1687several poll group related functions. 1688 1689A new flag, `create_only`, has been added to the `spdk_nvme_io_qpair_opts` structure. This flag 1690allows a user to call `spdk_nvme_ctrlr_get_default_io_qpair` without also connecting the qpair 1691within the context of that call. 1692 1693As a result of the `create_only` flag, two new API functions, `spdk_nvme_ctrlr_connect_io_qpair` 1694and `spdk_nvme_ctrlr_disconnect_io_qpair`, have been added to facilitate connecting newly created 1695qpairs (for example, after they have been added to a poll group) and disconnecting qpairs without 1696destroying them (for example to disconnect a qpair before migrating it to a new poll group and 1697reconnecting it). 1698 1699The functions `spdk_nvme_ctrlr_alloc_cmb_io_buffer` and `spdk_nvme_ctrlr_free_cmb_io_buffer` 1700have been changed to `spdk_nvme_ctrlr_map_cmb` and `spdk_nvme_ctrlr_unmap_cmb` respectively. 1701 1702An additional function, `spdk_nvme_ctrlr_reserve_cmb`, has been added to facilitate reserving 1703the entire size of the controller memory buffer for data transfer. 1704 1705### nvme_cuse 1706 1707`spdk_nvme_cuse_get_ctrlr_name` now takes two additional parameters, `char *name` which 1708stores the pointer to the controller name, and `size_t *size` which stores the length of 1709the name. The return type has also been changed from char * to int. 1710 1711`spdk_nvme_cuse_get_ns_name` now takes two additional parameters, `char *name` which 1712stores the pointer to the namespace name, and `size_t *size` which stores the length of 1713the name. The return type has also been changed from char * to int. 1714 1715### nvme_opal 1716 1717Several public OPAL structure definitions have been changed since the last release. 1718 1719### nvmf 1720 1721`spdk_nvmf_poll_group_destroy()` is now asynchronous and accepts a completion callback. 1722 1723The NVMe-oF target now creates a lightweight thread per poll group instead of assuming a pool 1724of lightweight threads already exist at start up time. A poll group is a collection of 1725unrelated NVMe-oF connections. Each poll group is only accessed from the associated 1726lightweight thread. 1727 1728A new struct, `spdk_nvmf_subsystem_listener`, has been added to encapsulate the subsystem specific 1729nature of a listener object. 1730 1731`spdk_nvmf_tgt_listen` no longer accepts a callback function or argument. It also returns an 1732int to indicate the status of the listen call. 1733 1734The execution of `spdk_nvme_poll_group_destroy` is now asynchronous and the function accepts 1735a cb_fn and cb_arg to call upon completion. 1736 1737The execution of `spdk_nvmf_subsystem_add_listener` is now asynchronous and the function accepts 1738a cb_fn and cb_arg to call upon completion. 1739 1740The `nvmf_transport.h` header has been made public to allow custom NVMe-oF transports to integrate 1741with NVMe-oF libraries without using internal APIs. 1742 1743### ocf 1744 1745Updated the OCF submodule to OCF v20.03 1746 1747New version of OCF comes with API changes and bug fixes 1748 1749### rpc 1750 1751A new RPC `thread_set_cpumask` has been added to set the cpumask of the thread 1752to the specified value. 1753 1754A new RPC `thread_get_pollers` has been added to retrieve pollers from SPDK threads. 1755 1756A new RPC `thread_get_io_channels` has been added to retrieve I/O channels from SPDK threads. 1757 1758A new RPC `bdev_rbd_resize` has been added to resize Ceph RBD bdevs. 1759 1760### sock 1761 1762The `spdk_sock_set_priority` function has been removed since the feature to set the sock priority 1763will be contained in two new functions, i.e., `spdk_sock_listen_ext` and `spdk_sock_connect_ext`. 1764Users may now specify the priority of the socket in the opts that they want to use. 1765 1766### spdk_top 1767 1768A new application, `spdk_top`, has been added which allows users to monitor resource consumption 1769by a running SPDK application. More information on this application can be found in 1770`app/spdk_top/README`. 1771 1772### thread 1773 1774A new function `spdk_thread_lib_init_ext` has been added, and the function 1775`spdk_thread_lib_init` has been deprecated. The user of `spdk_thread_lib_init_ext` is 1776expected to implement both functions `spdk_thread_op_fn` and `spdk_thread_op_supported_fn`. 1777`spdk_thread_op_supported_fn` is called to check whether the SPDK thread operation 1778is supported. `spdk_thread_op_fn` is called to execute the SPDK thread operation. 1779Current SPDK operation types are `SPDK_THREAD_OP_NEW` and `SPDK_THREAD_OP_RESCHED`. 1780The operation `SPDK_THREAD_OP_NEW` is called each time a new thread is created. 1781The operation `SPDK_THREAD_OP_RESCHED` is called when SPDK thread needs to be rescheduled. 1782 1783A unique ID has been added for each created SPDK thread, it is retrieved by a new function 1784`spdk_thread_get_id`, and the SPDK thread which has the specific ID is got by 1785a new function `spdk_thread_get_by_id`. 1786 1787A new function `spdk_thread_cpumask` has been added to set the current thread's cpumask 1788to the specified value. The function requires the operation `SPDK_THREAD_OP_RESCHED` 1789is supported. 1790 1791A new function `spdk_poller_register_named` has been added to set arbitrary name to the 1792created poller. If NULL, the name is set to the pointer of the poller function. 1793 1794The function `spdk_thread_poll` now measures run time per call correctly on multiple SPDK 1795threads configuration, and a new function `spdk_thread_get_last_tsc` has been added to use together. 1796 1797Voluntary termination of SPDK thread has been supported by refining the functions `spdk_thread_exit` 1798and `spdk_thread_poll`. 1799 1800### util 1801 1802New functions `spdk_sn32_lt` and `spdk_sn32_gt` have been added. They compare two sequence 1803numbers based on serial number arithmetic. 1804 1805### vhost 1806 1807Poll groups per session have been replaced by SPDK threads per vhost controller. 1808 1809### vmd 1810 1811A new function, `spdk_vmd_fini`, has been added. It releases all resources acquired by the VMD 1812library through the `spdk_vmd_init` call. 1813 1814## v20.01: Optimized thin provisioning, FTL bdev, VMD hot plug, FUSED support 1815 1816### bdev 1817 1818A new function, `spdk_bdev_set_timeout`, has been added to set per descriptor I/O timeouts. 1819 1820A new class of functions `spdk_bdev_compare*`, have been added to allow native bdev support 1821of block comparisons and compare-and-write. 1822 1823A new class of bdev events, `SPDK_BDEV_EVENT_MEDIA_MANAGEMENT`, has been added to allow bdevs 1824which expose raw media to alert all I/O channels of pending media management events. 1825 1826A new API was added `spdk_bdev_io_get_aux_buf` allowing the caller to request 1827an auxiliary buffer for its own private use. The API is used in the same manner that 1828`spdk_bdev_io_get_buf` is used and the length of the buffer is always the same as the 1829bdev_io primary buffer. 'spdk_bdev_io_put_aux_buf' frees the allocated auxiliary 1830buffer. 1831 1832### blobfs 1833 1834Added boolean return value for function spdk_fs_set_cache_size to indicate its operation result. 1835 1836Added `blobfs_set_cache_size` RPC method to set cache size for blobstore filesystem. 1837 1838### blobstore 1839 1840Added new `use_extent_table` option to `spdk_blob_opts` for creating blobs with Extent Table descriptor. 1841Using this metadata format, dramatically decreases number of writes required to persist each cluster allocation 1842for thin provisioned blobs. Extent Table descriptor is enabled by default. 1843See the [Blobstore Programmer's Guide](https://spdk.io/doc/blob.html#blob_pg_cluster_layout) for more details. 1844 1845### dpdk 1846 1847Updated DPDK submodule to DPDK 19.11. 1848 1849### env_dpdk 1850 1851`spdk_env_dpdk_post_init` now takes a boolean, `legacy_mem`, as an argument. 1852 1853A new function, `spdk_env_dpdk_dump_mem_stats`, prints information about the memory consumed by DPDK to a file specified by 1854the user. A new utility, `scripts/dpdk_mem_info.py`, wraps this function and prints the output in an easy to read way. 1855 1856### event 1857 1858The functions `spdk_reactor_enable_framework_monitor_context_switch()` and 1859`spdk_reactor_framework_monitor_context_switch_enabled()` have been changed to 1860`spdk_framework_enable_context_switch_monitor()` and 1861`spdk_framework_context_switch_monitor_enabled()`, respectively. 1862 1863### ftl 1864 1865All NVMe dependencies were removed from ftl library. 1866From now ftl library is using bdev_zone API. 1867bdev_ftl becomes virtual bdev. 1868 1869`ctrlr` and `trid` fields in `spdk_ftl_dev_init_opts` structure were replaced by 1870`base_bdev_desc`. 1871 1872`bdev_ftl_create` RPC method `trtype` and `traddr` parameters were replaced by `base_bdev` 1873parameter. 1874 1875`spdk_ftl_module_init` and `spdk_ftl_module_fini` functions were removed. 1876 1877`spdk_ftl_punit_range` and `ftl_module_init_opts` structures were removed. 1878 1879### isa-l 1880 1881Updated ISA-L submodule to commit f3993f5c0b6911 which includes implementation and 1882optimization for aarch64. 1883 1884Enabled ISA-L on aarch64 by default in addition to x86. 1885 1886### nvme 1887 1888`delayed_pcie_doorbell` parameter in `spdk_nvme_io_qpair_opts` was renamed to `delay_cmd_submit` 1889to allow reuse in other transports. 1890 1891Added RDMA WR batching to NVMf RDMA initiator. Send and receive WRs are chained together 1892and posted with a single call to ibv_post_send(receive) in the next call to qpair completion 1893processing function. Batching is controlled by 'delay_cmd_submit' qpair option. 1894 1895The NVMe-oF initiator now supports plugging out of tree NVMe-oF transports. In order 1896to facilitate this feature, several small API changes have been made: 1897 1898The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 1899A new function, `spdk_nvme_transport_available_by_name`, has been added. 1900A function table, `spdk_nvme_transport_ops`, and macro, `SPDK_NVME_TRANSPORT_REGISTER`, have been added which 1901enable registering out of tree transports. 1902 1903A new function, `spdk_nvme_ns_supports_compare`, allows a user to check whether a given namespace supports the compare 1904operation. 1905 1906A new family of functions, `spdk_nvme_ns_compare*`, give the user access to submitting compare commands to NVMe namespaces. 1907 1908A new function, `spdk_nvme_ctrlr_cmd_get_log_page_ext`, gives users more granular control over the command dwords sent in 1909log page requests. 1910 1911### nvmf 1912 1913Add SockPriority option in [Transport] section, this can be used for NVMe-oF target 1914on TCP transport to set sock priority for the incoming TCP connections. 1915 1916The NVMe-oF target now supports plugging out of tree NVMe-oF transports. In order 1917to facilitate this feature, several small API changes have been made: 1918 1919The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 1920`spdk_nvmf_tgt_get_transport`, `spdk_nvmf_transport_opts_init`, and `spdk_nvmf_transport_create` all have been 1921modified to take a string. 1922A function table, `spdk_nvmf_transport_ops`, and macro, `SPDK_NVMF_TRANSPORT_REGISTER`, have been added which 1923enable registering out of tree transports. 1924 1925Add `spdk_nvmf_tgt_stop_listen()` that can be used to stop listening for 1926incoming connections for specified target and trid. Listener is not stopped 1927implicitly upon destruction of a subsystem any more. 1928 1929A custom NVMe admin command handler has been added which allows the user to use the real drive 1930attributes from one of the target NVMe drives when reporting drive attributes to the initiator. 1931This handler can be enabled via the `nvmf_set_config` RPC. 1932Note: In a future version of SPDK, this handler will be enabled by default. 1933 1934The SPDK target and initiator both now include compare-and-write functionality with one caveat. If using the RDMA transport, 1935the target expects the initiator to send both the compare command and write command either with, or without inline data. The 1936SPDK initiator currently respects this requirement, but this note is included as a flag for other initiators attempting 1937compatibility with this version of SPDK. 1938 1939The `spdk_nvme_ctrlr_opts` struct has been extended with new field `transport_ack_timeout` which allows 1940to configure transport ACK timeout. Applicable for RDMA transport only. 1941 1942### rpc 1943 1944A new RPC, `bdev_zone_block_create`, enables creating an emulated zoned bdev on top of a standard block device. 1945 1946A new RPC, `bdev_ocssd_create`, enables creating an emulated zoned bdev on top of an Open Channel SSD. 1947 1948A new RPC, `blobfs_set_cache_size`, enables managing blobfs cache size. 1949 1950A new RPC, `env_dpdk_get_mem_stats`, has been added to facilitate reading DPDK related memory 1951consumption stats. Please see the env_dpdk section above for more details. 1952 1953A new RPC, `framework_get_reactors`, has been added to retrieve a list of all reactors. 1954 1955`bdev_ftl_create` now takes a `base_bdev` argument in lieu of `trtype`, `traddr`, and `punits`. 1956 1957`bdev_nvme_set_options` now allows users to disable I/O submission batching with the `-d` flag 1958 1959`bdev_nvme_cuse_register` now accepts a `name` parameter. 1960 1961`bdev_uring_create` now takes arguments for `bdev_name` and `block_size` 1962 1963`nvmf_set_config` now takes an argument to enable passthru of identify commands to base NVMe devices. 1964Please see the nvmf section above for more details. 1965 1966### scsi 1967 1968`spdk_scsi_lun_get_dif_ctx` now takes an additional argument of type `spdk_scsi_task`. 1969 1970### sock 1971 1972Added spdk_sock_writev_async for performing asynchronous writes to sockets. This call will 1973never return EAGAIN, instead queueing internally until the data has all been sent. This can 1974simplify many code flows that create pollers to continue attempting to flush writes 1975on sockets. 1976 1977Added `impl_name` parameter in spdk_sock_listen and spdk_sock_connect functions. Users may now 1978specify the sock layer implementation they'd prefer to use. Valid implementations are currently 1979"vpp" and "posix" and NULL, where NULL results in the previous behavior of the functions. 1980 1981### thread 1982 1983`spdk_thread_send_msg` now returns int indicating if the message was successfully 1984sent. 1985 1986A new function `spdk_thread_send_critical_msg`, has been added to support sending a single message from 1987a context that may be interrupted, e.g. a signal handler. 1988 1989Two new functions, `spdk_poller_pause`, and `spdk_poller_resume`, have been added to give greater control 1990of pollers to the application owner. 1991 1992### util 1993 1994`spdk_pipe`, a new utility for buffering data from sockets or files for parsing 1995has been added. The public API is available at `include/spdk/pipe.h`. 1996 1997### bdev 1998 1999Added spdk_bdev_io_get_nvme_fused_status function for translating bdev_io status to NVMe status 2000code for fused compare-and-write operation. 2001 2002Added spdk_bdev_get_acwu function for getting block device atomic compare and write unit size. 2003 2004## v19.10: Zoned bdev API, Opal bdev, NVMe character devices 2005 2006### rpc 2007 2008Many of SPDK's RPCs were renamed to be more consistent and intuitive in this release. 2009The old names will continue to function, but will display a deprecation warning. 2010 2011Added optional parameters '--arbitration-burst' and '--low/medium/high-priority-weight' to 2012'bdev_nvme_set_options' RPC method. 2013 2014Added optional parameter '--md-size' to 'construct_null_bdev' RPC method. 2015 2016Added optional parameters '--dif-type' and '--dif-is-head-of-md' to 'construct_null_bdev' 2017RPC method. 2018 2019Added `blobfs_detect` RPC method to detect whether a blobfs exists on given bdev. 2020 2021Added `blobfs_create` RPC method to build blobfs on given bdev. 2022 2023Added `blobfs_mount` RPC method to mount blobfs on given bdev to a host path by FUSE. 2024Then on the host path, user can directly do some file operations which will be mapped 2025to blobfs. 2026 2027### bdev 2028 2029Added new parameter `cdw0` to `spdk_bdev_io_complete_nvme_status()` and 2030`spdk_bdev_io_get_nvme_status()` that allows setting/getting 2031the NVMe completion queue DW0 entry. This allows vendor specific IO commands 2032to return command specific completion info back to the initiator. 2033 2034Added `spdk_bdev_get_write_unit_size()` function for retrieving required number 2035of logical blocks for write operation. 2036 2037New zone-related fields were added to the result of the `get_bdevs` RPC call: 2038 2039- `zoned`: indicates whether the device is zoned or a regular 2040 block device 2041- `zone_size`: number of blocks in a single zone 2042- `max_open_zones`: maximum number of open zones 2043- `optimal_open_zones`: optimal number of open zones 2044 2045The `zoned` field is a boolean and is always present, while the rest is only available for zoned 2046bdevs. 2047 2048A new `spdk_bdev_open_ext` function has been added and `spdk_bdev_open` function has been deprecated. 2049The new open function introduces requirement to provide callback function that will be called by 2050asynchronous event such as bdev removal. `spdk_bdev_open_ext` function takes bdev name as 2051an argument instead of bdev structure to avoid a race condition that can happen when the bdev 2052is being removed between a call to get its structure based on a name and actually openning it. 2053 2054New 'resize' event has been added to notify about change of block count property of block device. 2055Event is delivered only if block device was opened with `spdk_bdev_open_ext` function. 2056 2057### bdev zone 2058 2059Added new public header for zoned bdev. Zoned bdev is an extension 2060of the bdev interface. 2061 2062`spdk_bdev_get_zone_size()`, `spdk_bdev_get_max_open_zones()`, `spdk_bdev_get_optimal_open_zones()` 2063APIs were added for retrieving zoned device information. 2064`spdk_bdev_get_zone_info()` API was added for retrieving information about zones in zoned 2065device. 2066Added `spdk_bdev_zone_management()` API for changing zone state. 2067`spdk_bdev_zone_append()` and `spdk_bdev_zone_append_with_md()` APIs were added for 2068appending data to a zone. 2069Added `spdk_bdev_io_get_append location()` function for retrieving append location for I/O. 2070Added `spdk_bdev_is_zoned()` function for checking if bdev supports zoned namespace semantics. 2071 2072### bdev opal 2073 2074EXPERIMENTAL: A new opal bdev has been added to support management of 2075NVMe self-encrypting drives through the Opal specification. Users can 2076create opal bdevs from an NVMe namespace bdev, if the controller 2077containing that namespace supports Opal. Currently this is only 2078supported for namespace ID=1. The following RPCs have been added to 2079support Opal: `bdev_nvme_opal_init`, `bdev_nvme_opal_revert`, 2080`bdev_opal_create`, `bdev_opal_delete`, `bdev_opal_get_info`, 2081`bdev_opal_new_user`, `bdev_opal_set_lock_state`. 2082It does not yet support recreating the opal bdevs after application restart. 2083This bdev module should be considered very experimental, and the RPCs may 2084change significantly in future releases. 2085 2086### delay bdev 2087 2088The `bdev_delay_update_latency` has been added to allow users to update 2089a latency value for a given delay bdev. 2090 2091### compress bdev 2092 2093A new RPC `bdev_compress_get_orphans` has been added to list compress bdevs 2094that were not loaded due to a missing pm metadata file. In this state they 2095can only be deleted. 2096 2097### null bdev 2098 2099Metadata support has been added to Null bdev module. 2100 2101Protection information support has been added to Null bdev module. 2102 2103### nvme 2104 2105Added `no_shn_notification` to NVMe controller initialization options, users can enable 2106it for NVMe controllers. When the option is enabled, the controller will not do the 2107shutdown process and just disable the controller, users can start their application 2108later again to initialize the controller to the ready state. 2109 2110A controller flag `SPDK_NVME_CTRLR_WRR_SUPPORTED` was added to indicate the controller 2111can support weighted round robin arbitration feature with submission queue. 2112 2113Added `arbitration_burst` option for arbitration feature, and added three 2114`low/medium/high_priority_weight` options for weighted round robin arbitration. 2115 2116Added `spdk_nvme_ns_cmd_write_uncorrectable`. 2117 2118Added new error handling and reporting functionality. This includes several 2119new API functions to facilitate applications recovering when a qpair or 2120controller fails. 2121 2122`spdk_nvme_ctrlr_reconnect_io_qpair` - Reconnects a failed I/O qpair. 2123`spdk_nvme_ctrlr_set_trid` - Sets the trid of an existing controller. Can be used to 2124change the trid for failover cases. 2125`spdk_nvme_ctrlr_is_failed` - Returns the failed state of a controller. 2126`spdk_nvme_ctrlr_fail` - Forces a controller into a failed state. 2127 2128Modified the return behavior of several API functions to better indicate to 2129applications when a qpair is failed. This list of functions includes: 2130 2131`spdk_nvme_qpair_process_completions` 2132`spdk_nvme_ns_cmd_*` 2133`spdk_nvme_ctrlr_process_admin_completions` 2134`spdk_nvme_ctrlr_cmd_*` 2135 2136These functions now return -ENXIO when the qpair or controller on which they 2137operate is failed. 2138 2139EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux 2140kernel for controller as well as for namespace and process ioctl requests as usual 2141from linux environment. 2142 2143### nvmf 2144 2145The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts` 2146as its only parameter. This new structure contains the max_subsystems parameter previously 2147passed into that function. 2148 2149A new public API function `spdk_nvmf_get_tgt` has been added which allows users to 2150retrieve a pointer to an `spdk_nvmf_tgt` object by supplying its name. In the special 2151case where an RPC or application only creates a single target, this function can accept 2152a null name parameter and will return the only available target. 2153 2154The majority of the NVMe-oF RPCs now accept an optional tgt_name parameter. This will 2155allow those RPCs to work with applications that create more than one target. 2156 2157Three new NVMe-oF RPCs have been added `nvmf_create_target`, `nvmf_delete_target`, and 2158`nvmf_get_targets`. These new RPCs provide a basic interface for managing multiple target 2159objects. In SPDK the target object defines a unique discovery service. As of this release, 2160these RPCs are not intended to be used with the in-tree SPDK target applications, spdk_tgt and 2161nvmf_tgt, which use a single, global target structure. As such, they are not included in scripts/rpc.py 2162 2163Three new header functions have also been added to help deal with multiple targets. 2164`spdk_nvmf_tgt_get_name` takes a target pointer as an argument and returns its human readable name. 2165`spdk_nvmf_get_first_target` takes no arguments and returns the first target in the global list. 2166`spdk_nvmf_get_next_tgt` takes a target pointer as an argument and returns the next one in the global list. 2167 2168The `spdk_nvmf_tgt_accept` takes additional argument allowing to pass arbitrary context 2169information to the `new_qpair` callback. This will simplify the code when having multiple 2170nvmf targets or when retrieving the context information from globals is not suitable. 2171 2172### blobstore 2173 2174A new `spdk_bdev_create_bs_dev_from_desc` function has been added and `spdk_bdev_create_bs_dev` 2175function has been deprecated. 2176The new create function can cowork with `spdk_bdev_open_ext` function, which provides callback 2177function that will be called by asynchronous event such as bdev removal. 2178 2179### blobfs_bdev 2180 2181A new blobfs module `bdev` has been added to simplify the operations of blobfs on bdev. 2182 2183Function `spdk_blobfs_bdev_detect` is added to detect whether blobfs exists on the given block device. 2184 2185Function `spdk_blobfs_bdev_create` is added to create a blobfs on the given block device. 2186 2187Function `spdk_blobfs_bdev_mount` is added to mount a blobfs on the given block device to 2188a host path by FUSE. Then, a new thread is created dedicatedly for one mountpoint to handle 2189FUSE request by blobfs API. 2190 2191### build 2192 2193Option to build FUSE components into blobfs_bdev module for mounting a blobfs filesystem. 2194It requires the installation of libfuse3. By default, it is disabled. And it will be 2195enabled if run `./configure` with `--with-fuse` option. 2196 2197### iSCSI 2198 2199Portals may no longer be associated with a cpumask. The scheduling of 2200connections is moving to a more dynamic model. 2201 2202An new RPC `iscsi_portal_group_set_auth` has been added to set CHAP authentication 2203for discovery sessions specific for the existing iSCSI portal group. This RPC overwrites 2204the setting by the global parameters for the iSCSI portal group. 2205 2206### socket 2207 2208Added `spdk_sock_is_connected` to check whether the socket is currently connected. 2209`spdk_sock_group_poll` now returns number of events on success. 2210 2211### env 2212 2213Added `spdk_pci_device_unclaim()` function to cleanup pci claim file. 2214 2215### event 2216 2217`framework_start_init` RPC no longer stops the application on error during 2218initialization. 2219 2220### DPDK 2221 2222Updated DPDK submodule to DPDK 19.08. 2223 2224### ocf 2225 2226Updated OCF submodule to OCF v19.06 2227 2228Along with update, new cache mode 'write only' was added. 2229 2230New cache modes added to use via RPC, wi - write invalidate and wa - write around. 2231 2232New version of OCF provides fully asynchronous management API. 2233 2234## v19.07: NVMe-oF FC Transport, VMD, NVMe-oF Persistent reservations, Bdev I/O with separate metadata 2235 2236### ftl 2237 2238EXPERIMENTAL: Added ability to mirror writes to persistent write buffer cache 2239to allow for recovery from dirty shutdown event. 2240 2241Added handling of Asynchronous Nand Management Events (ANM). 2242 2243### vmd 2244 2245EXPERIMENTAL: Added Intel Volume Management Device (VMD) driver. VMD is an integrated 2246controller inside the CPU PCIe root complex. It enables virtual HBAs for the connected 2247NVMe SSDs. `spdk_vmd_init()` enumerates NVMe SSDs behind VMD device and hook them into 2248SPDK PCI subsystem. `spdk_nvme_probe()` or `spdk_nvme_connect()` can be used to connect 2249NVMe driver to the device located at the given transport ID. 2250 2251To obtain transport ID of NVMe SSD behind VMD `spdk_lspci` can be used. 2252 2253Current implementation does not support hotplug. 2254 2255### blobfs 2256 2257Blobfs file asynchronous operations were added to public APIs. 2258 2259### util 2260 2261A new file API `spdk_posix_file_load` was added to load file content into a data buffer. 2262 2263New APIs `spdk_dif_ctx_set_data_offset`, `spdk_dif_verify_stream`, 2264`spdk_dif_get_range_with_md`, `spdk_dif_get_length_with_md` have been added, 2265and existing APIs `spdk_dif_ctx_init`, `spdk_dif_set_md_interleave_iovs`, and 2266`spdk_dif_generate_stream` have been refined to insert or strip DIF into iovs 2267windows of any alignment. 2268 2269New APIs `spdk_dif_ctx_set_remapped_init_ref_tag`, `spdk_dif_remap_ref_tag`, 2270and `spdk_dix_remap_ref_tag` have been added to remap DIF reference tag. 2271 2272New APIs `spdk_dif_update_crc32c` and `spdk_dif_update_crc32c_stream` have been 2273added to compute CRC-32C checksum for extended LBA payload. 2274 2275Bdevperf and bdevio applications now support starting tests with application specific 2276RPCs. Please see helper Python scripts in their respective directories. 2277This is a move towards simpler RPC-only configuration for all main 2278and auxiliary applications. 2279 2280Legacy INI style configuration for SPDK applications will become deprecated in SPDK 19.10, 2281and removed in SPDK 20.01. Please consider moving to JSON-RPC configuration files and/or 2282RPC driven run-time configuration. 2283 2284### nvmf 2285 2286EXPERIMENTAL: A Fibre Channel transport that supports Broadcom HBAs has been 2287added. This depends on the FC HBA driver at 2288https://github.com/ecdufcdrvr/bcmufctdrvr. See [the documentation](https://spdk.io/doc/nvmf.html#nvmf_fc_transport) 2289for more information. 2290 2291Persistent reservation emulation has been added to the NVMe-oF target. Persistent reservation 2292state is stored in a JSON file on the local filesystem between target restart. To support this, 2293an optional parameter to the RPC method `nvmf_subsystem_add_ns` called `--ptpl-file` was added. 2294This allows the user to specify which file to store the persistent reservation state in. Note 2295that this is done per namespace. 2296 2297The c2h success optimization under which a command capsule response is not sent 2298for reads is turned on by default. A config knob was added to allow disabling 2299the optimization. This will mostly be used for integration testing with 5.0.x kernels 2300while some compatibility fixes make their way down the pipeline for 5.1.x kernels. 2301 2302The sock priority setting of the TCP connection owned by the tcp transport is added. It is 2303used to optimize the TCP connection performance under designated traffic classes. And the 2304priority is used to differentiate the sock priority between SPDK NVMe-oF TCP target application 2305and other TCP based applications. 2306 2307Shared receive queue can now be disabled even for NICs that support it using the 2308`nvmf_create_transport` RPC method parameter `no_srq`. The actual use of a shared 2309receive queue is predicated on hardware support when this flag is not used. 2310 2311spdk_nvmf_get_optimal_poll_group was added, which is used to return the optimal 2312poll group for the qpair. And `ConnectionScheduler` configuration is added into the 2313[Nvmf] section in etc/spdk/nvmf.conf.in to demonstrate how to configure the connection 2314scheduling strategy among different spdk threads. 2315 2316Added infrastructure to retrieve global and per poll group NVMf statistics. 2317 2318DIF strip and insert is now supported for TCP transport. When it is enabled, DIF 2319setting is not exposed to the NVMe-oF initiator, and DIF is attached into data 2320for write I/O and stripped from data for read I/O. 2321 2322Added a field `dif_insert_or_strip` to struct spdk_nvmf_transport_opts, and 2323updated the related rpc function nvmf_create_transport to make this 2324configurable parameter available to users. The `dif_insert_or_strip` is relevant 2325for TCP transport for now and used to configure the DIF strip and insert. 2326 2327Added infrastructure to retrieve NVMf transport statistics. 2328 2329### notify 2330 2331The function `spdk_notify_get_types()` and `spdk_notify_get_events()` were 2332renamed to `spdk_notify_foreach_type()` and `spdk_notify_foreach_event()`, 2333respectively. And update type name of callback accordingly. 2334 2335### bdev 2336 2337The format of the data returned by the get_bdevs_iostat RPC has changed to 2338make it easier to parse. It now returns an object with a "ticks" object 2339and "bdevs" array with the per-bdev statistics. 2340 2341A new bdev module `delay` has been added which simulates a drive latency when placed 2342on top of a Null bdev. This module is intended only for testing and can be created using 2343the new RPC `bdev_delay_create`. That RPC takes the name of the underlying bdev as well 2344as average and p99 latency arguments for both read and write operations. Average latency is 2345defined as a value close to what you would expect a perf tool such as FIO to report back as 2346the mean latency of all I/O submitted to the drive. p99 latency is defined as the value one 2347would expect the drive to see the slowest 1% of I/O report. For underlying drives with already 2348significant latency, the latency values provided to the drive will be additive. This should be 2349taken into account if trying to achieve an artificial latency on top of an nvme drive or aio device. 2350 2351DIF reference tag remapping is now supported for partition type virtual bdev 2352modules. When using partition type virtual bdevs, block address space is 2353remapped during I/O processing and DIF reference tag is remapped accordingly. 2354 2355Added spdk_bdev_*_with_md() functions allowing for IO with metadata being transferred in 2356separate buffer. To check support for separate metadata, use spdk_bdev_is_md_separate(). 2357 2358All bdevs now have a UUID. For devices whose backing hardware does not provide a UUID, 2359one is automatically generated. Across runs of SPDK, bdevs whose UUID is automatically 2360generated may change. 2361 2362A new virtual bdev module `compress` has been added to provide compression services on top of 2363a thinly provisioned logical volume. See documentation for complete details. 2364 2365### nvme 2366 2367Added an optional parameter `--io-queue-requests` to RPC `set_bdev_nvme_options`, which 2368can be used to change the number of requests allocated for one NVMe I/O queue. For 2369very big I/O size, e.g. 128MiB, with this option user will not get an error due to 2370limited requests in NVMe driver layer. 2371 2372Added spdk_nvme_ctrlr_get_transport_id() to get the transport ID from a 2373previously attached controller. 2374 2375Nvme Opal library spdk_opal_cmd deprecated. Adding separate command APIs. 2376NVMe Opal library add support for activating locking SP which will make the transaction 2377from "Manufactured-Inactive" state to "Manufactured" state. Upon successfully invoking 2378of this method, lock and unlock features will be enabled. 2379NVMe Opal library add support for locking/unlocking range and list locking range info. 2380NVMe opal library add support for multiuser. Admin can enable user and add user to specific 2381locking range and the user can lock/unlock his range. 2382 2383Added spdk_nvme_ctrlr_io_cmd_raw_no_payload_build() allowing a caller to pass 2384a completely formed command to an NVMe submission queue (buffer addresses and all). 2385This is supported on the PCIe transport only. 2386 2387Added spdk_nvme_get_ctrlr_registers() to return a pointer to the virtual address 2388of the NVMe controller registers. This is supported on the PCIe transport only. 2389 2390Added additional options to the spdk_nvme_ctrlr_alloc_qpair() option parameter 2391structure to allow caller to override the virtual and optionally physical address 2392of the submission and completion queue pair to be created. This is supported on 2393the PCIe transport only. 2394 2395Added `disable_error_logging` to struct spdk_nvme_ctrlr_opts, which disables 2396logging of failed requests. By default logging is enabled. 2397 2398Added spdk_nvme_qpair_print_command(), spdk_nvme_qpair_print_completion() and 2399spdk_nvme_cpl_get_status_string(). Allowing for easier display of error messages. 2400 2401Added support for NVMe Sanitize command. 2402 2403### env 2404 2405The parameter `free_space` has been added to spdk_ring_enqueue() to wait when 2406the ring is almost full and resume when there is enough space available in 2407the ring. 2408 2409A new API `spdk_mempool_lookup` has been added to lookup the memory pool created 2410by the primary process. 2411 2412Added spdk_pci_get_first_device() and spdk_pci_get_next_device() to allow 2413iterating over PCI devices detected by SPDK. Because of this, all SPDK APIs 2414to attach/detach PCI devices are no longer thread safe. They are now meant to 2415be called from only a single thread only, the same only that called spdk_env_init(). 2416This applies to the newly added APIs as well. 2417 2418### vpp 2419 2420SPDK now supports VPP version 19.04.2, up from VPP 18.01. 2421 2422VPP socket abstraction now uses VPP Session API, instead of VLC (VPP Communications Library). 2423This allows for better control over sessions and queues. 2424Please see VPP documentation for more details: 2425[VPP Host Stack](https://wiki.fd.io/view/VPP/HostStack) 2426 2427### sock 2428 2429Add spdk_sock_get_optimal_sock_group(), which returns the optimal sock group for 2430this socket. When a socket is created, it is often assigned to a sock group using 2431spdk_sock_group_add_sock so that a set of sockets can be polled more efficiently. 2432For some network devices, it is optimal to assign particular sockets to specific 2433sock groups. This API is intended to provide the user with that information. 2434 2435spdk_sock_group_get_ctx() was added to return the context of the spdk_sock_group. 2436spdk_sock_group_create() is updated to allow input the user provided ctx. 2437spdk_sock_set_priority() is added to set the priority of the socket. 2438 2439### rpc 2440 2441Added thread_get_stats RPC method to retrieve existing statistics. 2442 2443Added nvmf_get_stats RPC method to retrieve NVMf subsystem statistics. 2444 2445Response buffers for RPC requests are now always pre-allocated, which implies 2446that all spdk_jsonrpc_begin_result() calls always succeed and return a valid 2447buffer for JSON response. RPC calls no longer need to check if the buffer is 2448non-NULL. 2449 2450Added SPDK_RPC_REGISTER_ALIAS_DEPRECATED to help with deprecation process when 2451renaming existing RPC. First time a deprecated alias is used, it will print 2452a warning message. 2453 2454RPC `get_rpc_methods` was renamed `rpc_get_methods`. The old name is still usable, 2455but is now deprecated. 2456 2457### blobstore 2458 2459A snapshot can now be deleted if there is only a single clone on top of it. 2460 2461### build 2462 2463Preliminary support for cross compilation is now available. Targeting an older 2464CPU on the same architecture using your native compiler can be accomplished by 2465using the `--target-arch` option to `configure` as follows: 2466 2467~~~bash 2468./configure --target-arch=broadwell 2469~~~ 2470 2471Additionally, some support for cross-compiling to other architectures has been 2472added via the `--cross-prefix` argument to `configure`. To cross-compile, set CC 2473and CXX to the cross compilers, then run configure as follows: 2474 2475~~~bash 2476./configure --target-arch=aarm64 --cross-prefix=aarch64-linux-gnu 2477~~~ 2478 2479### vhost 2480 2481A security vulnerability has been identified and fixed in SPDK Vhost-SCSI target. 2482A malicious client (e.g. a virtual machine) could send a carefully prepared, 2483invalid I/O request to crash the entire SPDK process. All users of SPDK Vhost-SCSI 2484target are strongly recommended to update. All SPDK versions < 19.07 are affected. 2485 2486By default, SPDK will now rely on upstream DPDK's rte_vhost instead of its fork 2487located inside SPDK repo. The internal fork is still kept around to support older 2488DPDK versions, but is considered legacy and will be eventually removed. 2489 2490`configure` will now automatically use the upstream rte_vhost if the used DPDK 2491version is >= 19.05. 2492 2493spdk_vhost_init() is now asynchronous and accepts a completion callback. 2494 2495### iscsi target 2496 2497A security vulnerability has been identified and fixed in SPDK iSCSI target. 2498A malicious client (e.g. an iSCSI initiator) could send a carefully prepared, 2499invalid I/O request to crash the entire SPDK process. All users of SPDK iSCSI 2500target are strongly recommended to update. All SPDK versions < 19.07 are affected. 2501 2502### thread 2503 2504Exposed spdk_set_thread() in order for applications to associate 2505with SPDK thread when necessary. 2506 2507Added spdk_thread_destroy() to allow framework polling the thread to 2508release resources associated with that thread. 2509 2510## v19.04: Compression bdev, Notification library, NVMe Opal support 2511 2512### nvme 2513 2514Added asynchronous probe support. New APIs spdk_nvme_probe_async(), 2515spdk_nvme_connect_async() and spdk_nvme_probe_poll_async() were added to enable 2516this feature. spdk_nvme_probe_async() and spdk_nvme_connect_async() return a 2517context associated with the specified controllers. Users then call 2518spdk_nvme_probe_poll_async() until it returns 0, indicating that the operation 2519completed. 2520 2521A new qpair creation option, delay_pcie_doorbell, was added. This can be passed 2522to spdk_nvme_alloc_io_qpair(). This makes the I/O submission functions, such as 2523spdk_nvme_ns_writev(), skip ringing the submission queue doorbell. Instead the 2524doorbell will be rung as necessary inside spdk_nvme_qpair_process_completions(). 2525This can result in significantly fewer MMIO writes to the doorbell register 2526under heavy load, greatly improving performance. 2527 2528spdk_nvme_ctrlr_get_regs_cmbsz() was added to report the size of the controller 2529memory buffer, if available. 2530 2531spdk_nvme_ctrlr_get_flags() was added to return controller feature 2532flags. Two flags are currently tracked: 2533SPDK_NVME_CTRLR_SGL_SUPPORTED 2534SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED 2535 2536The NVMe hotplug poller is now able to detach devices hot-removed from the system 2537via `/sys/bus/pci/devices/<bdf>/remove` and `/sys/bus/pci/devices/<bdf>/driver/unbind`. 2538 2539Opal support was added for scan, take ownership, revert TPer, and dumping device 2540info. The nvme_manage tool can be used to perform these operations. The public 2541API functions are spdk_nvme_ctrlr_security_receive() and 2542spdk_nvme_ctrlr_security_send(). This module should be considered experimental 2543pending additional features and tests. 2544 2545The NVMe-oF initiator is now able to transfer IO whose size is more than 128KiB 2546if the corresponding NVMe-oF target allows. 2547 2548### raid 2549 2550Added new strip_size_kb rpc param on create to replace the more ambiguous 2551strip_size. The strip_size rpc param is deprecated. 2552 2553Changed the raid bdev product_name from "Pooled Device" to "Raid Volume" 2554 2555### thread 2556 2557Added spdk_thread_has_pollers() function to verify if there are any registered 2558pollers to be run on the thread. Added spdk_thread_is_idle() function to check 2559if there are any scheduled operations to be performed on the thread at given 2560time. 2561 2562spdk_thread_create() now takes an optional CPU affinity mask that is passed to 2563the scheduler. 2564 2565spdk_thread_lib_init() now takes an optional context size. For each thread 2566created, an additional region of memory of this size will be allocated. A 2567pointer to this region of memory can be obtained by calling 2568spdk_thread_get_ctx(). The inverse operation is also available via 2569spdk_thread_get_from_ctx(). 2570 2571spdk_thread_poll() now optionally accepts the current time, in ticks. This can 2572be used to avoid some calls to spdk_get_ticks() internally. 2573 2574spdk_app_start() now only accepts a single context argument. 2575 2576### bdev 2577 2578An new API `spdk_bdev_get_data_block_size` has been added to get size of data 2579block except for metadata. 2580 2581spdk_vbdev_register() has been deprecated. spdk_bdev_register() should be used 2582instead. 2583 2584A mechanism for acquiring and releasing data buffers from bdev modules, used 2585to perform zero copy operations, was added. 2586 2587New APIs spdk_bdev_get_md_size(), spdk_bdev_is_md_interleaved(), spdk_bdev_get_dif_type(), 2588spdk_bdev_is_dif_head_of_md(), and spdk_bdev_is_dif_check_enabled() have been 2589added to get metadata and DIF settings. 2590 2591Bdevs claimed by the `examine_config` callback will be now further examined in the 2592`examine_disk` callback. 2593 2594spdk_bdev_io_get_io_channel() was added as a convenient way to get an io_channel 2595from a bdev_io. 2596 2597### ocf 2598 2599Updated OCF submodule to OCF v19.3.2 2600 2601Added support for many-to-one configuration for OCF bdev. 2602Multiple core devices can now be cached on single cache device. 2603 2604Added persistent metadata support, allowing to restore cache state after shutdown. 2605During start of SPDK application, the devices are examined and if OCF metadata 2606is present - appropriate OCF bdevs will be recreated. 2607 2608Added Write-Back mode support. In this mode, data is first written to 2609caching device and periodically synchronized to the core devices. 2610Dirty data is saved as persistent metadata on cache device, 2611allowing for safe restore during application restart. 2612For more details please see OCF documentation: 2613[OpenCAS cache configuration](https://open-cas.github.io/cache_configuration.html) 2614 2615### NVMe-oF Target 2616 2617Support for per-device shared receive queues in the RDMA transport has been added. 2618It is enabled by default for any device that supports it. 2619 2620The size of a shared receive queue is defined by transport configuration file parameter 2621`MaxSRQDepth` and `nvmf_create_transport` RPC method parameter `max_srq_depth`. 2622Default size is 4096. 2623 2624Add model number as parameter to construct_nvmf_subsystem (-d option), 2625rather than using hardcoded define. 2626 2627DIF passthrough feature has been added. DIF setting of the allocated bdevs is 2628exposed to the NVMe-oF initiator and data with DIF from the NVMe-oF initiator is 2629passed through to the allocated bdevs. 2630 2631### env 2632 2633The `phys_addr` parameter in spdk_malloc() and spdk_zmalloc() has been deprecated. 2634For retrieving physical addresses, spdk_vtophys() should be used instead. 2635 2636spdk_realloc() has been added to reallocate DMA/shared memory. 2637 2638spdk_pci_device_is_removed() has been added to let the upper-layer SPDK drivers know 2639that device has a pending external hotremove request. 2640 2641spdk_env_fini() and spdk_env_dpdk_post_fini() were added to release any resources 2642allocated by spdk_env_init() or spdk_env_dpdk_post_init() respectively. It is expected 2643that common usage of those functions is to call them just before terminating the process. 2644 2645Added spdk_iommu_is_enabled() to report if SPDK application is using IOMMU for DMA. 2646 2647### DPDK 2648 2649Dropped support for DPDK 17.07 and earlier, which SPDK won't even compile with right now. 2650 2651Updated DPDK submodule to DPDK 19.02. 2652 2653### rpc 2654 2655New `get_spdk_version` RPC method is introduced to get version info of the running SPDK application. 2656 2657The `start_nbd_disk` RPC method now take nbd_device as an optional parameter. If nbd_device 2658is specified, use that specified nbd device. If it's not specified, pick available one. 2659 2660### iSCSI target 2661 2662DIF strip and insert is now supported. DIF settings are not exposed to the iSCSI initiator. 2663DIF is attached into data for write I/O and stripped from data for read I/O. 2664 2665### vhost 2666 2667Added experimental support for running with the external, upstream rte_vhost library. 2668This can be enabled by configuring SPDK with an `--without-internal-vhost-lib` flag. 2669The minimum supported rte_vhost version (DPDK version) is 19.05-rc1. 2670 2671As a result of fuzz testing, a lot of data races in vhost-scsi LUN hotplug path were identified and 2672fixed. Those data races could have potentially resulted in SPDK crashes, RPC hangs, or memory leaks 2673if Vhost-SCSI LUN hotplug RPCs were executed while connected VMs were in the middle of restarting. 2674 2675The SCSI target id in `add_vhost_scsi_lun` RPC is now optional. If `-1` is passed, the first 2676unoccupied target id will be used. 2677 2678### AIO 2679 2680AIO bdev module can now reap I/O completions directly from userspace, significantly improving 2681the overall performance. 2682 2683### blobfs 2684 2685Synchronous IO operations no longer use spdk_io_channel, but instead use 2686spdk_fs_thread_ctx. The behavior is otherwise identical. 2687 2688### OCF 2689 2690Added support for caching multiple bdevs using a single bdev as a cache. 2691 2692### notify 2693 2694Added the notify library that provides a high performance local event bus 2695between libraries. Example usage was added to bdev module, which reports 2696notifications for added and removed bdevs via RPC. 2697 2698### sock 2699 2700Added new API spdk_sock_readv() to the sock library for performing vectored 2701reads. 2702 2703### event 2704 2705The function spdk_subsystem_init() no longer requires spdk_event as an argument. 2706 2707Changed API of spdk_subsystem_config_json() to no longer be asynchronous. 2708 2709### io_uring 2710 2711A bdev module that issues I/O to kernel block devices using the new io_uring Linux kernel 2712API was added. This module requires liburing. 2713 2714### build 2715 2716Options to easily compile with profile guided optimization have been added to 2717`configure`. To use profile guided optimization with SPDK, run 2718`./configure --with-pgo-capture`, build SPDK, then run a workload of your 2719choosing. Then, simply run `./configure --with-pgo-enable` and recompile to 2720build using the generated profile data. Profile guided optimization can yield 2721very large performance improvements, especially on GCC 8 and clang 7. This may 2722be combined with link time optimization which has been available under the 2723`--enable-lto` configure option for several releases. 2724 2725### compression bdev/reduce library 2726 2727Added "reduce" block compression scheme based on using SSDs for storing 2728compressed blocks of storage and persistent memory for metadata. Please see 2729[compression](https://spdk.io/doc/bdev.html) for more details. 2730 2731## v19.01: NVMe-oF TCP/IP Transport, Open Channel SSD Flash Translation Layer, Caching bdev based on OCF, ISA-L Support, DIF/DIX library 2732 2733### ocf bdev 2734 2735New virtual bdev module based on [Open CAS Framework](https://open-cas.github.io/) has been added. 2736This module allows for the use of one bdev to act as a high performance cache in front of another bdev. 2737Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_cas) for more details. 2738Only write through mode is currently supported and this feature is considered experimental. 2739 2740### event framework 2741 2742For `spdk_app_parse_args`, add return value to the callback which parses application 2743specific command line parameters to protect SPDK applications from crashing by invalid 2744values from user input. 2745 2746By default, all SPDK applications will now reserve all hugepages at runtime. The pre-reserved 2747memory size can be still set with `-s` or `--mem-size` option, although the default value 2748was reduced down to 0. 2749 2750A custom hugetlbfs directory can now be specified via spdk_app_opts. 2751This can be used to configure hugepages with different sizes, a different size limit, 2752or different access permissions than the system's default hugepage pool. 2753SPDK applications can specify a custom hugetlbfs mount with the `--huge-dir` option. 2754 2755### environment 2756 2757spdk_vtophys() has been refactored to accept length of the translated region as a new 2758parameter. The function will now update that parameter with the largest possible value 2759for which the memory is contiguous in the physical memory address space. 2760 2761The following functions were removed: 2762 2763- spdk_pci_nvme_device_attach() 2764- spdk_pci_nvme_enumerate() 2765- spdk_pci_ioat_device_attach() 2766- spdk_pci_ioat_enumerate() 2767- spdk_pci_virtio_device_attach() 2768- spdk_pci_virtio_enumerate() 2769 2770They were replaced with generic spdk_pci_device_attach() and spdk_pci_enumerate() which 2771require a new spdk_pci_driver object to be provided. It can be one of the following: 2772 2773- spdk_pci_nvme_get_driver() 2774- spdk_pci_ioat_get_driver() 2775- spdk_pci_virtio_get_driver() 2776 2777spdk_pci_hook_device() and spdk_pci_unhook_device() were added. Those allow adding a virtual 2778spdk_pci_device into the SPDK PCI subsystem. A virtual device calls provided callbacks for 2779each BAR mapping request or PCI config access. It's attachable with spdk_pci_device_attach() 2780or spdk_pci_enumerate() like any other device. 2781 2782A new spdk_pause() function was added to pause CPU execution for an implementation specific 2783amount of time. Quoting from DPDK function this is based on: "This call is intended for 2784tight loops which poll a shared resource or wait for an event. A short pause within the loop 2785may reduce the power consumption." 2786 2787A new public header file env_dpdk.h has been introduced, and function spdk_env_dpdk_post_init 2788is added into it. If user is using DPDK, and already called rte_eal_init, then include 2789include/spdk/env_dpdk.h, and call spdk_env_dpdk_post_init() instead of spdk_env_init. 2790 2791ISA-L has been added as an SPDK submodule. ISA-L is enabled by default on x86 architecture 2792to accelerate algorithms such as CRC for iSCSI and NVMe-oF. Users may still disable ISA-L 2793by explicitly passing --without-isal to the configure script. 2794 2795### util 2796 2797A new uuid API `spdk_uuid_copy` was added to make a copy of the source uuid. 2798 2799An new parameter `init_crc` representing the initial CRC value was added to 2800`spdk_crc16_t10dif`. The parameter can be used to calculate a CRC value spanning 2801multiple separate buffers. 2802 2803New DIF APIs were added to generate and verify DIF by byte granularity for both DIF and DIX 2804formats. Among them, DIF with copy APIs will be usable to emulate DIF operations such as DIF 2805insert and strip. 2806 2807Added `spdk_strtol` and `spdk_strtoll` to provide additional error checking around `strtol` 2808and `strtoll`. 2809 2810Added `spdk_sprintf_append_realloc` and `spdk_vsprintf_append_realloc` for appending a string 2811with automatic buffer re-allocation. 2812 2813### nvme 2814 2815Wrapper functions spdk_nvme_ctrlr_security_send() and spdk_nvme_ctrlr_security_receive() are 2816introduced to support further security protocol development. 2817 2818admin_timeout_ms was added to NVMe controller initialization options, users 2819can change the default value when probing a controller. 2820 2821Add two new fields "header_digest" and "data_digest" in struct spdk_nvme_ctrlr_opts, 2822it will be used to enable the digest support for the NVMe/TCP transport. 2823 2824Add a new TCP/IP transport(located in lib/nvme/nvme_tcp.c) in nvme driver. With 2825this new transport, it can be used to connect the NVMe-oF target with the 2826same TCP/IP support. 2827 2828Added API, spdk_nvme_ctrlr_is_discovery(), to indicate whether the ctrlr 2829arg refers to a Discovery Controller or not. 2830 2831Added an API function `spdk_nvme_host_id_parse` and corresponding object `spdk_nvme_host_id` 2832for parsing host address and host service ID arguments on a per connection basis. 2833 2834The RPC `construct_nvme_bdev` now allows a user to specify a source address and service id for the host to 2835use when connecting to the controller backing the NVMe bdev. 2836 2837### NVMe-oF Target 2838 2839The `spdk_nvmf_tgt_opts` struct has been deprecated in favor of `spdk_nvmf_transport_opts`. 2840Users will no longer be able to specify target wide I/O parameters. `spdk_nvmf_tgt_listen` 2841will also no longer implicitly initialize a transport with the default target options (since 2842there are none). Instead, a user must manually instantiate the transport with `spdk_nvmf_transport_create` 2843prior to calling `spdk_nvmf_tgt_listen`. 2844 2845Related to the previous change, the rpc `set_nvmf_target_options` has been renamed to 2846`set_nvmf_target_max_subsystems` to indicate that this is the only target option available for the user to edit. 2847 2848Added fields `num_shared_buffers` and `buf_cache_size` in struct spdk_nvmf_transport_opts, 2849and also updated the related rpc function nvmf_create_transport, to make this 2850configurable parameter available to users. The `num_shared_buffers` is used to 2851configure the shared buffer numbers of the transport used by RDMA or TCP transport. 2852`buf_cache_size` configures number of shared buffers to cache per poll group. 2853 2854### nvmf 2855 2856Add a new TCP/IP transport (located in lib/nvmf/tcp.c). With this transport, 2857the SPDK NVMe-oF target can have a new transport, and can serve the NVMe-oF 2858protocol via TCP/IP from the host. 2859 2860Added optional mechanism to modify the RDMA transport's behavior when creating protection domains and registering memory. 2861By default, the RDMA transport will use the ibverbs library to create protection domains and register memory. 2862Using `spdk_nvme_rdma_init_hooks` will subvert that and use an existing registration. 2863 2864### bdev 2865 2866Added `enable_bdev_histogram` and `get_bdev_histogram` RPC commands to allow gathering latency data for specified bdev. 2867Please see [documentation](https://spdk.io/doc/bdev.html#rpc_bdev_histogram) for more details. 2868 2869Added `required_alignment` field to `spdk_bdev`, that specifies an alignment requirement for data buffers associated with an spdk_bdev_io. 2870Bdev layer will automatically double buffer any spdk_bdev_io that violates this alignment, before the spdk_bdev_io is submitted to the bdev module. 2871 2872On shutdown, bdev unregister now proceeds in top-down fashion, with 2873claimed bdevs skipped (these will be unregistered later, when virtual 2874bdev built on top of the respective base bdev unclaims it). This 2875allows virtual bdevs to be shut down cleanly as opposed to the 2876previous behavior that didn't differentiate between hotremove and 2877planned shutdown. 2878 2879The `delete_bdev` RPC is now deprecated. Users should instead use the specific deletion RPC 2880for the bdev type to be removed (i.e. delete_malloc_bdev). 2881 2882Added support for separate bandwidth rate limits for read and write to QoS in bdev layer. 2883 2884Bdev I/O statistics now track unmap operations. 2885 2886### logical volumes 2887 2888Logical volume bdev can now be marked as read only using `set_read_only_lvol_bdev` RPC. 2889This allows for basing clones on top of lvol_bdev without first creating a snapshot. 2890 2891Added option to change method for data erasure when deleting lvol or resizing down. 2892Default of unmapping clusters can now be changed to writing zeroes or no operation. 2893 2894Added option to change method for erasing data region on lvol store creation. 2895Default of unmapping can now be changed to writing zeroes or no operation. 2896 2897### log 2898 2899"trace flags" are now referred to as "log flags" in the SPDK log API. The 2900set_trace_flag, clear_trace_flag and get_trace_flags RPCs are now deprecated, 2901and set_log_flag, clear_log_flag and get_log_flags RPCs have been added. 2902 2903### trace 2904 2905New `trace_record` application was added. It can be used to poll spdk trace shm file and 2906append any new trace entries into another specified file. This can help retain those entries 2907that would otherwise be overwritten in the shm file. See 2908[Capturing sufficient trace events](https://spdk.io/doc/nvmf_tgt_tracepoints.html#capture_trace_events) 2909for more details. 2910 2911Number of trace entries in circular buffer per lcore can now be assigned by starting SPDK app 2912with argument "--num-trace-entries <NUM>" provided. 2913 2914New `get_tpoint_group_mask` RPC was added to get current tpoint_group_mask, and 2915each tpoint group status. 2916New `enable_tpoint_group` and `disable_tpoint_group` RPC were added to enable or 2917disable a specific tpoint group. 2918 2919### ftl 2920 2921EXPERIMENTAL: Added basic flash translation layer module allowing for using Open Channel SSDs as 2922block devices. The module is split into the library (located in lib/ftl) and bdev_ftl 2923(lib/bdev/ftl). See the [documentation](https://spdk.io/doc/ftl.html) for more details. 2924 2925### vhost 2926 2927A security vulnerability has been identified and fixed in the SPDK vhost target. A malicious 2928vhost client (i.e. virtual machine) could carefully construct a circular descriptor chain which 2929would result in a partial denial of service in the SPDK vhost target. These types of descriptor 2930chains are now properly detected by the vhost target. All SPDK vhost users serving untrusted 2931vhost clients are strongly recommended to upgrade. (Reported by Dima Stepanov and Evgeny 2932Yakovlev.) 2933 2934Vhost SCSI and Vhost Block devices can now accept multiple connections on the same socket file. 2935Each connection (internally called a vhost session) will have access to the same storage, but 2936will use different virtqueues, different features and possibly different memory. 2937 2938### vhost scsi 2939 2940SCSI target hotremove can now be performed even without the VIRTIO_SCSI_F_HOTPLUG feature negotiated. 2941Regardless of VIRTIO_SCSI_F_HOTPLUG support, the hotremoval will be still reported through SCSI sense codes. 2942 2943### DPDK 2944 2945DPDK submodule was updated to DPDK 18.11. Note that SPDK does not fully leverage the new 2946multi-process device hotplug yet and continues to work the same way it always did. 2947 2948Dropped support for DPDK 16.07 and earlier, which SPDK won't even compile with right now. 2949 2950### RPC 2951 2952The following RPC commands deprecated in the previous release are now removed: 2953 2954- construct_virtio_user_scsi_bdev 2955- construct_virtio_pci_scsi_bdev 2956- construct_virtio_user_blk_bdev 2957- construct_virtio_pci_blk_bdev 2958- remove_virtio_scsi_bdev 2959- construct_nvmf_subsystem 2960 2961### Miscellaneous 2962 2963The configure options `--with-raid` and `--without-raid` that were deprecated in the previous 2964release are now removed. 2965 2966### nbd 2967 2968Starting nbd using `spdk_nbd_start` is now performed asynchronously. 2969 2970### net framework 2971 2972Net framework initialization and finish is now done asynchronously. 2973 2974### rpc 2975 2976Added `spdk_rpc_is_method_allowed` function for checking whether method is permitted in a given state. 2977Added `spdk_rpc_get_state` to check current state of RPC server. 2978RPC `wait_subsystem_init` has been added to allow clients to block untill all subsystems are initialized. 2979 2980### json rpc 2981 2982JSON RPC client is now running in non-blocking mode. Requests are sent and received during spdk_jsonrpc_client_poll. 2983JSON RPC server can now receive a callback on connection termination or server shutdown using `spdk_jsonrpc_conn_add_close_cb` 2984and `spdk_jsonrpc_conn_del_close_cb`. 2985 2986## v18.10: Dynamic memory allocation, Crypto Virtual Bdev, jsonrpc-client, SPDKCLI iSCSI and NVMe-oF support 2987 2988### nvme 2989 2990spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive() 2991were added to support sending or receiving security protocol data to or from 2992nvme controller. 2993 2994spdk_nvme_ns_get_extended_sector_size() was added. This function includes 2995the metadata size per sector (if any). spdk_nvme_ns_get_sector_size() still 2996returns only the data size per sector, not including metadata. 2997 2998New `send_nvme_cmd` RPC was added to allow sending NVMe commands directly to NVMe controller. 2999See the [send_nvme_cmd](http://spdk.io/doc/jsonrpc.html#rpc_send_nvme_cmd) documentation 3000for more details. 3001 3002### Build System 3003 3004New `configure` options, `--with-shared` and `--without-shared` 3005[default], provide the capability to build, or not, SPDK shared libraries. 3006This includes the single SPDK shared lib encompassing all of the SPDK 3007static libs as well as individual SPDK shared libs corresponding to 3008each of the SPDK static ones. Although the production of the shared 3009libs conforms with conventional version naming practices, such naming 3010does not at this time confer any SPDK ABI compatibility claims. 3011 3012### bdev 3013 3014spdk_bdev_alias_del_all() was added to delete all alias from block device. 3015 3016A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev 3017Framework. The module initially uses a software AESNI CBC cipher with experimental support for the 3018Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work 3019may include additional ciphers as well as consideration for authentication. 3020 3021The RAID virtual bdev module is now always enabled by default. The configure --with-raid and 3022--without-raid options are now ignored and deprecated and will be removed in the next release. 3023 3024Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer. 3025See the new [set_bdev_qos_limit](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit) 3026documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at 302718.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both 3028bandwidth and IOPS limits. 3029 3030spdk_bdev_config_json() and corresponding `get_bdevs_config` RPC was removed. 3031 3032### Environment Abstraction Layer and Event Framework 3033 3034The size parameter of spdk_mem_map_translate is now a pointer. This allows the 3035function to report back the actual size of the translation relative to the original 3036request made by the user. 3037 3038A new structure spdk_mem_map_ops has been introduced to hold memory map related 3039callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc 3040in lieu of the notify callback. 3041 3042### DPDK 18.08 3043 3044The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically 3045utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1. 3046 3047Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup, 3048but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try 3049to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using 3050only as many hugepages as it is really needed. 3051 3052Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed 3053to be physically contiguous. 3054 3055### I/OAT 3056 3057I/OAT driver can now reinitialize I/OAT channels after encountering DMA errors. 3058 3059### iscsi target 3060 3061Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC 3062method for CHAP authentication in discovery sessions have been changed to 3063align with `construct_target_node` RPC method. Old names are still usable 3064but will be removed in future release. 3065 3066`set_iscsi_discovery_auth` and `set_iscsi_target_node_auth` RPC methods have 3067been added to set CHAP authentication for discovery sessions and existing 3068target nodes, respectively. 3069 3070The SPDK iSCSI target supports an AuthFile which can be used to load CHAP 3071shared secrets when the iSCSI target starts. SPDK previously provided a 3072default location for this file (`/usr/local/etc/spdk/auth.conf`) if none was 3073specified. This default has been removed. Users must now explicitly specify 3074the location of this file to load CHAP shared secrets from a file, or use 3075the related iSCSI RPC methods to add them at runtime. 3076 3077### iscsi initiator 3078 3079The SPDK iSCSI initiator is no longer considered experimental and becomes 3080a first-class citizen among bdev modules. The basic usage has been briefly 3081described in the bdev user guide: [iSCSI bdev](https://spdk.io/doc/bdev.html#bdev_config_iscsi) 3082 3083### Miscellaneous 3084 3085The SPDK application framework can now parse long name command line parameters. 3086Most single-character parameters have a long name equivalent now. See the 3087[Command Line Parameters](https://spdk.io/doc/app_overview.html) documentation 3088for details or use the `--help` command line parameter to list all available 3089params. 3090 3091bdevperf `-s` param (io size) was renamed to `-o` as `-s` had been already 3092used by existing apps for memory size. 3093 3094bdevio can now accept all SPDK command line parameters. The config now has to 3095be provided with `-c` or `--config` param. 3096 3097The following ioat/perf and nvme/perf parameters were renamed as well: 3098 `-s` (io size) to `-o` 3099 `-d` (mem size) to `-s` 3100 3101The ReactorMask config file parameter has been deprecated. Users should 3102use the -m or --cpumask command line option to specify the CPU core mask 3103for the application. 3104 3105Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt 3106and vhost. Config file pathnames may now only be specified using the 3107-c command line option. 3108 3109Users may no longer set DPDK_DIR in their environment to specify the 3110location of the DPDK installation used to build SPDK. Using DPDK_DIR 3111has not been the documented nor recommended way to specify the DPDK 3112location for several releases, but removing it ensures no unexpected 3113surprises for users who may have DPDK_DIR defined for other reasons. 3114Users should just use the "configure" script to specify the DPDK 3115location before building SPDK. 3116 3117Although we know that many developers still use Python 2 we are officially 3118switching to Python3 with requirement that all new code must be valid also 3119for Python 2 up to the EOL which is year 2020. 3120 3121Invoking interpreter explicitly is forbidden for executable scripts. There 3122is no need to use syntax like "python ./scripts/rpc.py". All executable 3123scripts must contain proper shebang pointing to the right interpreter. 3124Scripts without shebang mustn't be executable. 3125 3126A Python script has been added to enable conversion of old INI config file 3127to new JSON-RPC config file format. This script can be found at 3128scripts/config_converter.py. Example how this script can be used: 3129~~~{.sh} 3130cat old_format.ini | scripts/config_converter.py > new_json_format.json 3131~~~ 3132 3133### Sock 3134 3135Two additional parameters were added to spdk_sock_get_addr() for the server 3136port and client port. These parameters are named "sport" and "cport" 3137respectively. 3138 3139### Virtio 3140 3141The following RPC commands have been deprecated: 3142 3143- construct_virtio_user_scsi_bdev 3144- construct_virtio_pci_scsi_bdev 3145- construct_virtio_user_blk_bdev 3146- construct_virtio_pci_blk_bdev 3147- remove_virtio_scsi_bdev 3148 3149The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev` 3150command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev` 3151was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI 3152devices. 3153 3154### Blobfs 3155 3156spdk_file_get_id() returning unique ID for the file was added. 3157 3158### JSON 3159 3160Added jsonrpc-client C library intended for issuing RPC commands from applications. 3161 3162Added API enabling iteration over JSON object: 3163 3164- spdk_json_find() 3165- spdk_json_find_string() 3166- spdk_json_find_array() 3167- spdk_json_object_first() 3168- spdk_json_array_first() 3169- spdk_json_next() 3170 3171### Blobstore 3172 3173Blobstore I/O operations are now based on io_units, instead of blobstore page size. 3174The io_unit size is now the same as the underlying block device's block size. 3175Logical volumes built on a block device with 512B block size can now be used as boot devices 3176in QEMU. 3177 3178### SPDKCLI 3179 3180The SPDKCLI interactive command tool for managing SPDK is no longer considered experimental. 3181Support for the iSCSI and NVMe-oF targets has been added. 3182 3183## v18.07: Raid, Infrastructure Improvements, Bug Fixes 3184 3185### bdev 3186 3187A new public header file bdev_module.h has been introduced to facilitate the 3188development of new bdev modules. This header includes an interface for the 3189spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of 3190multiple virtual bdevs on top of a single base bdev and should act as the 3191primary API for module authors. 3192 3193spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide 3194options. 3195 3196A mechanism for handling out of memory condition errors (ENOMEM) returned from 3197I/O submission requests at the bdev layer has been added. See 3198spdk_bdev_queue_io_wait(). 3199 3200The spdk_bdev_get_io_stat() function now returns cumulative totals instead of 3201resetting on each call. This allows multiple callers to query I/O statistics 3202without conflicting with each other. Existing users will need to adjust their 3203code to record the previous I/O statistics to calculate the delta between calls. 3204 3205I/O queue depth tracking and samples options have been added. See 3206spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and 3207spdk_bdev_set_qd_sampling_period(). 3208 3209### RAID module 3210 3211A new bdev module called "raid" has been added as experimental module which 3212aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note 3213that vhost will not work with this module because it does not yet have support 3214for multi-element io vectors. 3215 3216### Log 3217 3218The debug log component flag available on several SPDK applications has been 3219renamed from `-t` to `-L` to prevent confusion with tracepoints and to allow the 3220option to be added to tools that already use `-t` to mean something else. 3221 3222### Blobstore 3223 3224A new function, spdk_bs_dump(), has been added that dumps all of the contents of 3225a blobstore to a file pointer. This includes the metadata and is very useful for 3226debugging. 3227 3228Two new operations have been added for thin-provisioned blobs. 3229spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions 3230of the blob and populate them with the correct data by reading from the backing 3231blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate 3232clusters that correspond to data in the blob's immediate parent. Clusters 3233allocated to grandparents or that aren't allocated at all will remain 3234thin-provisioned. 3235 3236### BlobFS 3237 3238Changed the return type of spdk_file_truncate() from void to int to allow the 3239propagation of `ENOMEM` errors. 3240 3241### NVMe Driver 3242 3243The new API functions spdk_nvme_qpair_add_cmd_error_injection() and 3244spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error 3245emulation. Users can set a specified command to fail with a particular error 3246status. 3247 3248Changed the name `timeout_sec` parameter to `timeout_us` in 3249spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from 3250uint32_t to uint64_t. This will give users more fine-grained control over the 3251timeout period. 3252 3253Basic support for Open Channel SSDs was added. See nvme_ocssd.h 3254 3255### NVMe Over Fabrics 3256 3257The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback 3258as a parameter. 3259 3260spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs. 3261 3262spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed 3263number of namespaces for a given subsystem. 3264 3265### Build System 3266 3267The build system now generates a combined shared library (libspdk.so) that may 3268be used in place of the individual static libraries (libspdk_*.a). The combined 3269library includes all components of SPDK and is intended to make linking against 3270SPDK easier. The static libraries are also still provided for users that prefer 3271to link only the minimal set of components required. 3272 3273### git pre-commit and pre-push hooks 3274 3275The pre-commit hook will run `scripts/check_format.sh` and verify there are no 3276formating errors before allowing `git commit` to run. The pre-push hook runs 3277`make CONFIG_WERROR=y` with and without `CONFIG_DEBUG=y` using both the gcc and 3278clang compiler before allowing `git push` to run. Following each DEBUG build 3279`test/unit/unittest.sh` is run and verified. Results are recorded in the 3280`make.log` file. 3281 3282To enable type: 'git config core.hooksPath .githooks'. To override after 3283configuration use the `git --no-verify` flag. 3284 3285### RPC 3286 3287The `start_nbd_disk` RPC method now returns the path to the kernel NBD device node 3288rather than always returning `true`. 3289 3290### DPDK 18.05 3291 3292The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports 3293dynamic memory allocation, but due to some issues found after the DPDK 18.05 release, 3294that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use 3295the legacy memory allocation model. The plan is to enable dynamic memory allocation 3296after the DPDK 18.08 release which should fix these issues. 3297 3298### Environment Abstraction Layer and Event Framework 3299 3300The spdk_mem_map_translate() function now takes a size parameter to indicate the size of 3301the memory region. This can be used by environment implementations to validate the 3302requested translation. 3303 3304The I/O Channel implementation has been moved to its own library - lib/thread. The 3305public API that was previously in spdk/io_channel.h is now in spdk/thread.h The 3306file spdk/io_channel.h remains and includes spdk/thread.h. 3307 3308spdk_reactor_get_tsc_stats was added to return interesting statistics for each 3309reactor. 3310 3311### IOAT 3312 3313IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable 3314option with "Yes" in `[Ioat]` section of the configuration file. The Disable option is 3315now deprecated and will be removed in a future release. 3316 3317## v18.04: Logical Volume Snapshot/Clone, iSCSI Initiator, Bdev QoS, VPP Userspace TCP/IP 3318 3319### vhost 3320 3321The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the 3322DPDK rte_vhost vulnerability [CVE-2018-1059](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1059). 3323Please see this [security advisory](https://access.redhat.com/security/cve/cve-2018-1059) 3324for additional information on the DPDK vulnerability. 3325 3326Workarounds have been added to ensure vhost compatibility with QEMU 2.12. 3327 3328EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the 3329[vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3330 3331### Unified Target Application 3332 3333A new unified SPDK target application, `spdk_tgt`, has been added. This application combines the 3334functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target, 3335and vhost target. The new application can be managed through the existing configuration file and 3336[JSON-RPC](http://www.spdk.io/doc/jsonrpc.html) methods. 3337 3338### Env 3339 3340spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk(). 3341 3342New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added. 3343These new functions have a `flags` parameter that allows the user to specify whether the allocated 3344memory needs to be suitable for DMA and whether it should be shared across processes with the same 3345shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will 3346eventually be deprecated and removed. 3347 3348### Bdev 3349 3350A new optional bdev module interface function, `init_complete`, has been added to notify bdev modules 3351when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require 3352notification that the set of initialization examine() calls is complete. 3353 3354The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with 3355the spdk_bdev_get_uuid() function. 3356 3357Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the 3358[set_bdev_qos_limit_iops](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit_iops) documentation 3359for more details. 3360 3361### RPC 3362 3363The `[Rpc]` configuration file section, which was deprecated in v18.01, has been removed. 3364Users should switch to the `-r` command-line parameter instead. 3365 3366The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as 3367needed; previously, the response was limited to 32 kilobytes. 3368 3369### SPDKCLI 3370 3371EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available. 3372See the [SPDKCLI](http://www.spdk.io/doc/spdkcli.html) documentation for more details. 3373 3374### NVMe Driver 3375 3376EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is 3377experimental pending a functional allocator to free and reallocate CMB buffers. 3378 3379spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available. 3380 3381New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns() 3382have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if 3383a namespace ID is active. 3384 3385### NVMe-oF Target 3386 3387Namespaces may now be assigned unique identifiers via new optional `eui64` and `nguid` parameters 3388to the `nvmf_subsystem_add_ns` RPC method. Additionally, the NVMe-oF target automatically exposes 3389the backing bdev's UUID as the namespace UUID when available. 3390 3391spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion. 3392 3393### Blobstore 3394 3395A number of functions have been renamed: 3396 3397- spdk_bs_io_write_blob() => spdk_blob_io_write() 3398- spdk_bs_io_read_blob() => spdk_blob_io_read() 3399- spdk_bs_io_writev_blob() => spdk_blob_io_writev() 3400- spdk_bs_io_readv_blob() => spdk_blob_io_readv() 3401- spdk_bs_io_unmap_blob() => spdk_blob_io_unmap() 3402- spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes() 3403 3404The old names still exist but are deprecated. They will be removed in the v18.07 release. 3405 3406spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O 3407are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still 3408required to sync the updated metadata to disk. 3409 3410### Logical Volumes 3411 3412A new `destroy_lvol_bdev` RPC method to delete logical volumes has been added. 3413 3414Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination. 3415 3416New Snapshot and Clone functionalities have been added. User may create Snapshots of existing Lvols 3417and Clones of existing Snapshots. 3418See the [lvol snapshots](http://www.spdk.io/doc/logical_volumes.html#lvol_snapshots) documentation 3419for more details. 3420 3421Resizing logical volumes is now supported via the `resize_lvol_bdev` RPC method. 3422 3423### Lib 3424 3425A set of changes were made in the SPDK's lib code altering 3426instances of calls to `exit()` and `abort()` to return a failure instead 3427wherever reasonably possible. 3428 3429spdk_app_start() no longer exit()'s on an internal failure, but 3430instead returns a non-zero error status. 3431 3432spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid 3433option, but instead returns SPDK_APP_PARSE_ARGS_HELP and 3434SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS 3435on success. 3436 3437spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07. 3438 3439### I/O Channels 3440 3441The prototype for spdk_poller_fn() has been modified; it now returns a value indicating 3442whether or not the poller did any work. Existing pollers will need to be updated to 3443return a value. 3444 3445### iSCSI Target 3446 3447The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace 3448TCP/IP stack. See the [iSCSI VPP documentation](http://www.spdk.io/doc/iscsi.html#vpp) for more 3449details. 3450 3451### iSCSI initiator 3452 3453An iSCSI initiator bdev module has been added to SPDK. This module should be considered 3454experimental pending additional features and tests. More details can be found in 3455lib/bdev/iscsi/README. 3456 3457### PMDK 3458 3459The persistent memory (PMDK) bdev module is now enabled using `--with-pmdk` instead of 3460`--with-nvml`. This reflects the renaming of the persistent memory library from NVML to 3461PMDK. 3462 3463### Virtio Block driver 3464 3465A userspace driver for Virtio Block devices has been added. It was built on top of the 3466[Virtio](http://www.spdk.io/doc/virtio.html) library and can be managed similarly to 3467the Virtio SCSI driver. See the 3468[Virtio Block](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_blk) reference for 3469more information. 3470 3471### Virtio with 2MB hugepages 3472 3473The previous 1GB hugepage limitation has now been lifted. A new `-g` command-line option 3474enables SPDK Virtio to work with 2MB hugepages. 3475See [2MB hugepages](http://www.spdk.io/doc/virtio.html#virtio_2mb) for details. 3476 3477## v18.01: Blobstore Thin Provisioning 3478 3479### Build System 3480 3481The build system now includes a `make install` rule, including support for the common 3482`DESTDIR` and `prefix` variables as used in other build systems. Additionally, the prefix 3483may be set via the configure `--prefix` option. Example: `make install prefix=/usr`. 3484 3485### RPC 3486 3487A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/tmp/spdk.sock. 3488A -r option command line option has been added to enable an alternative UNIX domain socket location, 3489or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file 3490section is now deprecated and will be removed in the v18.04 release. 3491 3492### I/O Channels 3493 3494spdk_poller_register() and spdk_poller_unregister() were moved from the event 3495framework (include/spdk/event.h) to the I/O channel library 3496(include/spdk/io_channel.h). This allows code that doesn't depend on the event 3497framework to request registration and unregistration of pollers. 3498 3499spdk_for_each_channel() now allows asynchronous operations during iteration. 3500Instead of immediately continuing the interaction upon returning from the iteration 3501callback, the user must call spdk_for_each_channel_continue() to resume iteration. 3502 3503### Block Device Abstraction Layer (bdev) 3504 3505The poller abstraction was removed from the bdev layer. There is now a general purpose 3506abstraction for pollers available in include/spdk/io_channel.h 3507 3508### Lib 3509 3510A set of changes were made in the SPDK's lib code altering, 3511instances of calls to `exit()` and `abort()` to return a failure instead 3512wherever reasonably possible. This has resulted in return type changes of 3513the API for: 3514 3515- spdk_env_init() from type `void` to `int`. 3516- spdk_mem_map_init() from type `void` to `int`. 3517 3518Applications making use of these APIs should be modified to check for 3519a non-zero return value instead of relying on them to fail without return. 3520 3521### NVMe Driver 3522 3523SPDK now supports hotplug for vfio-attached devices. But there is one thing keep in mind: 3524Only physical removal events are supported; removing devices via the sysfs `remove` file will not work. 3525 3526### NVMe-oF Target 3527 3528Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available 3529cores round-robin. The "Core" option in the configuration file has been removed. 3530 3531### Blobstore 3532 3533A number of functions have been renamed: 3534 3535- spdk_bs_md_resize_blob() => spdk_blob_resize() 3536- spdk_bs_md_sync_blob() => spdk_blob_sync_md() 3537- spdk_bs_md_close_blob() => spdk_blob_close() 3538- spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names() 3539- spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value() 3540- spdk_blob_md_set_xattr() => spdk_blob_set_xattr() 3541- spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr() 3542- spdk_bs_md_create_blob() => spdk_bs_create_blob() 3543- spdk_bs_md_open_blob() => spdk_bs_open_blob() 3544- spdk_bs_md_delete_blob() => spdk_bs_delete_blob() 3545- spdk_bs_md_iter_first() => spdk_bs_iter_first() 3546- spdk_bs_md_iter_next() => spdk_bs_iter_next() 3547 3548The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument 3549rather than struct spdk_blob **. 3550 3551The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument 3552rather than struct spdk_blob **. 3553 3554Thin provisioning support has been added to the blobstore. It can be enabled by setting the 3555`thin_provision` flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext(). 3556 3557### NBD device 3558 3559The NBD application (test/lib/bdev/nbd) has been removed; Same functionality can now be 3560achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method. 3561See the [GPT](http://www.spdk.io/doc/bdev.html#bdev_config_gpt) documentation for more details. 3562 3563### FIO plugin 3564 3565SPDK `fio_plugin` now supports FIO 3.3. The support for previous FIO 2.21 has been dropped, 3566although it still remains to work for now. The new FIO contains huge amount of bugfixes and 3567it's recommended to do an update. 3568 3569### Virtio library 3570 3571Previously a part of the bdev_virtio module, now a separate library. Virtio is now available 3572via `spdk_internal/virtio.h` file. This is an internal interface to be used when implementing 3573new Virtio backends, namely Virtio-BLK. 3574 3575### iSCSI 3576 3577The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated 3578to an epoll/kqueue descriptor on the master core when idle. 3579 3580## v17.10: Logical Volumes 3581 3582### New dependencies 3583 3584libuuid was added as new dependency for logical volumes. 3585 3586libnuma is now required unconditionally now that the DPDK submodule has been updated to DPDK 17.08. 3587 3588### Block Device Abstraction Layer (bdev) 3589 3590An [fio](http://github.com/axboe/fio) plugin was added that can route 3591I/O to the bdev layer. See the [plugin documentation](https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin/) 3592for more information. 3593 3594spdk_bdev_unmap() was modified to take an offset and a length in bytes as 3595arguments instead of requiring the user to provide an array of SCSI 3596unmap descriptors. This limits unmaps to a single contiguous range. 3597 3598spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out. 3599If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using 3600write commands. 3601 3602New API functions that accept I/O parameters in units of blocks instead of bytes 3603have been added: 3604 3605- spdk_bdev_read_blocks(), spdk_bdev_readv_blocks() 3606- spdk_bdev_write_blocks(), spdk_bdev_writev_blocks() 3607- spdk_bdev_write_zeroes_blocks() 3608- spdk_bdev_unmap_blocks() 3609 3610The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be 3611retried later. 3612 3613### Linux AIO bdev 3614 3615The AIO bdev now allows the user to override the auto-detected block size. 3616 3617### NVMe driver 3618 3619The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field. 3620NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take 3621into account when splitting I/O requests. 3622 3623The HotplugEnable option in `[Nvme]` sections of the configuration file is now 3624"No" by default. It was previously "Yes". 3625 3626The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the 3627NVMe Controller associated with a given namespace. 3628 3629The NVMe library now allows the user to specify a host identifier when attaching 3630to a controller. The host identifier is used as part of the Reservations feature, 3631as well as in the NVMe-oF Connect command. The default host ID is also now a 3632randomly-generated UUID, and the default host NQN uses the host ID to generate 3633a UUID-based NQN. 3634 3635spdk_nvme_connect() was added to allow the user to connect directly to a single 3636NVMe or NVMe-oF controller. 3637 3638### NVMe-oF Target (nvmf_tgt) 3639 3640The NVMe-oF target no longer requires any in-capsule data buffers to run, and 3641the feature is now entirely optional. Previously, at least 4 KiB in-capsule 3642data buffers were required. 3643 3644NVMe-oF subsystems have a new configuration option, AllowAnyHost, to control 3645whether the host NQN whitelist is enforced when accepting new connections. 3646If no Host options have been specified and AllowAnyHost is disabled, the 3647connection will be denied; this is a behavior change from previous releases, 3648which allowed any host NQN to connect if the Host list was empty. 3649AllowAnyHost is disabled by default. 3650 3651NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number 3652of namespaces per subsystem is no longer limited. 3653 3654The NVMe-oF target now supports the Write Zeroes command. 3655 3656### Environment Abstraction Layer 3657 3658A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide 3659additional clarity when constructing spdk_mempools. Previously, -1 could be 3660passed and the library would choose a reasonable default, but this new value 3661makes it explicit that the default is being used. 3662 3663### Blobstore 3664 3665The blobstore super block now contains a bstype field to identify the type of the blobstore. 3666Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load(). 3667 3668spdk_bs_destroy() was added to allow destroying blobstore on device 3669with an initialized blobstore. 3670 3671spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable 3672scattered payloads. 3673 3674A CLI tool for blobstore has been added, allowing basic operations through either command 3675line or shell interface. See the [blobcli](https://github.com/spdk/spdk/tree/master/examples/blob/cli) 3676documentation for more details. 3677 3678### Event Framework 3679 3680The ability to set a thread name, previously only used by the reactor code, is 3681now part of the spdk_thread_allocate() API. Users may specify a thread name 3682which will show up in tools like `gdb`. 3683 3684### Log 3685 3686The spdk_trace_dump() function now takes a new parameter to allow the caller to 3687specify an output file handle (stdout or stderr, for example). 3688 3689### Logical Volumes 3690 3691Logical volumes library built on top of SPDK blobstore has been added. 3692It is possible to create logical volumes on top of other devices using RPC. 3693 3694See the [logical volumes](http://www.spdk.io/doc/logical_volumes.html) documentation for more information. 3695 3696### Persistent Memory 3697 3698A new persistent memory bdev type has been added. 3699The persistent memory block device is built on top of [libpmemblk](http://pmem.io/nvml/libpmemblk/). 3700It is possible to create pmem devices on top of pmem pool files using RPC. 3701 3702See the [Pmem Block Device](http://www.spdk.io/doc/bdev.html#bdev_config_pmem) documentation for more information. 3703 3704### Virtio SCSI driver 3705 3706A userspace driver for Virtio SCSI devices has been added. 3707The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application. 3708 3709See the [Virtio SCSI](http://www.spdk.io/doc/virtio.html) documentation and 3710[Getting Started](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_scsi) guide for more information. 3711 3712### Vhost target 3713 3714The vhost target application now supports live migration between QEMU instances. 3715 3716## v17.07: Build system improvements, userspace vhost-blk target, and GPT bdev 3717 3718### Build System 3719 3720A `configure` script has been added to simplify the build configuration process. 3721The existing CONFIG file and `make CONFIG_...` options are also still supported. 3722Run `./configure --help` for information about available configuration options. 3723 3724A DPDK submodule has been added to make building SPDK easier. If no `--with-dpdk` 3725option is specified to configure, the SPDK build system will automatically build a 3726known-good configuration of DPDK with the minimal options enabled. See the Building 3727section of README.md for more information. 3728 3729A [Vagrant](https://www.vagrantup.com/) setup has been added to make it easier to 3730develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant 3731section of README.md for more information. 3732 3733### Userspace vhost-blk target 3734 3735The vhost library and example app have been updated to support the vhost-blk 3736protocol in addition to the existing vhost-scsi protocol. 3737See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3738 3739### Block device abstraction layer (bdev) 3740 3741A GPT virtual block device has been added, which automatically exposes GPT partitions 3742with a special SPDK-specific partition type as bdevs. 3743See the [GPT bdev documentation](http://www.spdk.io/doc/bdev.md#bdev_config_gpt) for 3744more information. 3745 3746### NVMe driver 3747 3748The NVMe driver has been updated to support recent Intel SSDs, including the Intel® 3749Optane™ SSD DC P4800X series. 3750 3751A workaround has been added for devices that failed to recognize register writes 3752during controller reset. 3753 3754The NVMe driver now allocates request tracking objects on a per-queue basis. The 3755number of requests allowed on an I/O queue may be set during `spdk_nvme_probe()` by 3756modifying `io_queue_requests` in the opts structure. 3757 3758The SPDK NVMe `fio_plugin` has been updated to support multiple threads (`numjobs`). 3759 3760spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override 3761controller-level options for each individual I/O queue pair. 3762Existing callers with qprio == 0 can be updated to: 3763~~~ 3764... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0); 3765~~~ 3766Callers that need to specify a non-default qprio should be updated to: 3767~~~ 3768struct spdk_nvme_io_qpair_opts opts; 3769spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts)); 3770opts.qprio = SPDK_NVME_QPRIO_...; 3771... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts)); 3772~~~ 3773 3774### Environment Abstraction Layer 3775 3776The environment abstraction layer has been updated to include several new functions 3777in order to wrap additional DPDK functionality. See `include/spdk/env.h` for the 3778current set of functions. 3779 3780### SPDK Performance Analysis with Intel® VTune™ Amplifier 3781 3782Support for SPDK performance analysis has been added to Intel® VTune™ Amplifier 2018. 3783 3784This analysis provides: 3785 3786- I/O performance monitoring (calculating standard I/O metrics like IOPS, throughput, etc.) 3787- Tuning insights on the interplay of I/O and compute devices by estimating how many cores 3788 would be reasonable to provide for SPDK to keep up with a current storage workload. 3789 3790See the VTune Amplifier documentation for more information. 3791 3792## v17.03: Blobstore and userspace vhost-scsi target 3793 3794### Blobstore and BlobFS 3795 3796The blobstore is a persistent, power-fail safe block allocator designed to be 3797used as the local storage system backing a higher-level storage service. 3798See the [blobstore documentation](http://www.spdk.io/doc/blob.html) for more details. 3799 3800BlobFS adds basic filesystem functionality like filenames on top of the blobstore. 3801This release also includes a RocksDB Env implementation using BlobFS in place of the 3802kernel filesystem. 3803See the [BlobFS documentation](http://www.spdk.io/doc/blobfs.html) for more details. 3804 3805### Userspace vhost-scsi target 3806 3807A userspace implementation of the QEMU vhost-scsi protocol has been added. 3808The vhost target is capable of exporting SPDK bdevs to QEMU-based VMs as virtio devices. 3809See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3810 3811### Event framework 3812 3813The overhead of the main reactor event loop was reduced by optimizing the number of 3814calls to spdk_get_ticks() per iteration. 3815 3816### NVMe library 3817 3818The NVMe library will now automatically split readv/writev requests with scatter-gather 3819lists that do not map to valid PRP lists when the NVMe controller does not natively 3820support SGLs. 3821 3822The `identify` and `perf` NVMe examples were modified to add a consistent format for 3823specifying remote NVMe over Fabrics devices via the `-r` option. 3824This is implemented using the new `spdk_nvme_transport_id_parse()` function. 3825 3826### iSCSI Target 3827 3828The [Nvme] section of the configuration file was modified to remove the `BDF` directive 3829and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 3830devices can now be specified as the backing block device. A script to generate an 3831entire [Nvme] section based on the local NVMe devices attached was added at 3832`scripts/gen_nvme.sh`. 3833 3834### NVMe-oF Target 3835 3836The [Nvme] section of the configuration file was modified to remove the `BDF` directive 3837and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 3838devices can now be specified as the backing block device. A script to generate an 3839entire [Nvme] section based on the local NVMe devices attached was added at 3840`scripts/gen_nvme.sh`. 3841 3842## v16.12: NVMe over Fabrics host, hotplug, and multi-process 3843 3844### NVMe library 3845 3846The NVMe library has been changed to create its own request memory pool rather than 3847requiring the user to initialize the global `request_mempool` variable. Apps can be 3848updated by simply removing the initialization of `request_mempool`. Since the NVMe 3849library user no longer needs to know the size of the internal NVMe request 3850structure to create the pool, the `spdk_nvme_request_size()` function was also removed. 3851 3852The `spdk_nvme_ns_cmd_deallocate()` function was renamed and extended to become 3853`spdk_nvme_ns_cmd_dataset_management()`, which allows access to all of the NVMe 3854Dataset Management command's parameters. Existing callers can be updated to use 3855`spdk_nvme_ns_cmd_dataset_management()` with `SPDK_NVME_DSM_ATTR_DEALLOCATE` as the 3856`type` parameter. 3857 3858The NVMe library SGL callback prototype has been changed to return virtual addresses 3859rather than physical addresses. Callers of `spdk_nvme_ns_cmd_readv()` and 3860`spdk_nvme_ns_cmd_writev()` must update their `next_sge_fn` callbacks to match. 3861 3862The NVMe library now supports NVMe over Fabrics devices in addition to the existing 3863support for local PCIe-attached NVMe devices. For an example of how to enable 3864NVMe over Fabrics support in an application, see `examples/nvme/identify` and 3865`examples/nvme/perf`. 3866 3867Hot insert/remove support for NVMe devices has been added. To enable NVMe hotplug 3868support, an application should call the `spdk_nvme_probe()` function on a regular 3869basis to probe for new devices (reported via the existing `probe_cb` callback) and 3870removed devices (reported via a new `remove_cb` callback). Hotplug is currently 3871only supported on Linux with the `uio_pci_generic` driver, and newly-added NVMe 3872devices must be bound to `uio_pci_generic` by an external script or tool. 3873 3874Multiple processes may now coordinate and use a single NVMe device simultaneously 3875using [DPDK Multi-process Support](http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html). 3876 3877### NVMe over Fabrics target (`nvmf_tgt`) 3878 3879The `nvmf_tgt` configuration file format has been updated significantly to enable 3880new features. See the example configuration file `etc/spdk/nvmf.conf.in` for 3881more details on the new and changed options. 3882 3883The NVMe over Fabrics target now supports virtual mode subsystems, which allow the 3884user to export devices from the SPDK block device abstraction layer as NVMe over 3885Fabrics subsystems. Direct mode (raw NVMe device access) is also still supported, 3886and a single `nvmf_tgt` may export both types of subsystems simultaneously. 3887 3888### Block device abstraction layer (bdev) 3889 3890The bdev layer now supports scatter/gather read and write I/O APIs, and the NVMe 3891blockdev driver has been updated to support scatter/gather. Apps can use the 3892new scatter/gather support via the `spdk_bdev_readv()` and `spdk_bdev_writev()` 3893functions. 3894 3895The bdev status returned from each I/O has been extended to pass through NVMe 3896or SCSI status codes directly in cases where the underlying device can provide 3897a more specific status code. 3898 3899A Ceph RBD (RADOS Block Device) blockdev driver has been added. This allows the 3900`iscsi_tgt` and `nvmf_tgt` apps to export Ceph RBD volumes as iSCSI LUNs or 3901NVMe namespaces. 3902 3903### General changes 3904 3905`libpciaccess` has been removed as a dependency and DPDK PCI enumeration is 3906used instead. Prior to DPDK 16.07 enumeration by class code was not supported, 3907so for earlier DPDK versions, only Intel SSD DC P3x00 devices will be discovered 3908by the NVMe library. 3909 3910The `env` environment abstraction library has been introduced, and a default 3911DPDK-based implementation is provided as part of SPDK. The goal of the `env` 3912layer is to enable use of alternate user-mode memory allocation and PCI access 3913libraries. See `doc/porting.md` for more details. 3914 3915The build process has been modified to produce all of the library files in the 3916`build/lib` directory. This is intended to simplify the use of SPDK from external 3917projects, which can now link to SPDK libraries by adding the `build/lib` directory 3918to the library path via `-L` and linking the SPDK libraries by name (for example, 3919`-lspdk_nvme -lspdk_log -lspdk_util`). 3920 3921`nvmf_tgt` and `iscsi_tgt` now have a JSON-RPC interface, which allows the user 3922to query and modify the configuration at runtime. The RPC service is disabled by 3923default, since it currently does not provide any authentication or security 3924mechanisms; it should only be enabled on systems with controlled user access 3925behind a firewall. An example RPC client implemented in Python is provided in 3926`scripts/rpc.py`. 3927 3928## v16.08: iSCSI target, NVMe over Fabrics maturity 3929 3930This release adds a userspace iSCSI target. The iSCSI target is capable of exporting 3931NVMe devices over a network using the iSCSI protocol. The application is located 3932in app/iscsi_tgt and a documented configuration file can be found at etc/spdk/spdk.conf.in. 3933 3934This release also significantly improves the existing NVMe over Fabrics target. 3935 3936- The configuration file format was changed, which will require updates to 3937 any existing nvmf.conf files (see `etc/spdk/nvmf.conf.in`): 3938 - `SubsystemGroup` was renamed to `Subsystem`. 3939 - `AuthFile` was removed (it was unimplemented). 3940 - `nvmf_tgt` was updated to correctly recognize NQN (NVMe Qualified Names) 3941 when naming subsystems. The default node name was changed to reflect this; 3942 it is now "nqn.2016-06.io.spdk". 3943 - `Port` and `Host` sections were merged into the `Subsystem` section 3944 - Global options to control max queue depth, number of queues, max I/O 3945 size, and max in-capsule data size were added. 3946 - The Nvme section was removed. Now a list of devices is specified by 3947 bus/device/function directly in the Subsystem section. 3948 - Subsystems now have a Mode, which can be Direct or Virtual. This is an attempt 3949 to future-proof the interface, so the only mode supported by this release 3950 is "Direct". 3951- Many bug fixes and cleanups were applied to the `nvmf_tgt` app and library. 3952- The target now supports discovery. 3953 3954This release also adds one new feature and provides some better examples and tools 3955for the NVMe driver. 3956 3957- The Weighted Round Robin arbitration method is now supported. This allows 3958 the user to specify different priorities on a per-I/O-queue basis. To 3959 enable WRR, set the `arb_mechanism` field during `spdk_nvme_probe()`. 3960- A simplified "Hello World" example was added to show the proper way to use 3961 the NVMe library API; see `examples/nvme/hello_world/hello_world.c`. 3962- A test for measuring software overhead was added. See `test/lib/nvme/overhead`. 3963 3964## v16.06: NVMf userspace target 3965 3966This release adds a userspace NVMf (NVMe over Fabrics) target, conforming to the 3967newly-released NVMf 1.0/NVMe 1.2.1 specification. The NVMf target exports NVMe 3968devices from a host machine over the network via RDMA. Currently, the target is 3969limited to directly exporting physical NVMe devices, and the discovery subsystem 3970is not supported. 3971 3972This release includes a general API cleanup, including renaming all declarations 3973in public headers to include a `spdk` prefix to prevent namespace clashes with 3974user code. 3975 3976- NVMe 3977 - The `nvme_attach()` API was reworked into a new probe/attach model, which 3978 moves device detection into the NVMe library. The new API also allows 3979 parallel initialization of NVMe controllers, providing a major reduction in 3980 startup time when using multiple controllers. 3981 - I/O queue allocation was changed to be explicit in the API. Each function 3982 that generates I/O requests now takes a queue pair (`spdk_nvme_qpair *`) 3983 argument, and I/O queues may be allocated using 3984 `spdk_nvme_ctrlr_alloc_io_qpair()`. This allows more flexible assignment of 3985 queue pairs than the previous model, which only allowed a single queue 3986 per thread and limited the total number of I/O queues to the lowest number 3987 supported on any attached controller. 3988 - Added support for the Write Zeroes command. 3989 - `examples/nvme/perf` can now report I/O command latency from the 3990 the controller's viewpoint using the Intel vendor-specific read/write latency 3991 log page. 3992 - Added namespace reservation command support, which can be used to coordinate 3993 sharing of a namespace between multiple hosts. 3994 - Added hardware SGL support, which enables use of scattered buffers that 3995 don't conform to the PRP list alignment and length requirements on supported 3996 NVMe controllers. 3997 - Added end-to-end data protection support, including the ability to write and 3998 read metadata in extended LBA (metadata appended to each block of data in the 3999 buffer) and separate metadata buffer modes. 4000 See `spdk_nvme_ns_cmd_write_with_md()` and `spdk_nvme_ns_cmd_read_with_md()` 4001 for details. 4002- IOAT 4003 - The DMA block fill feature is now exposed via the `ioat_submit_fill()` 4004 function. This is functionally similar to `memset()`, except the memory is 4005 filled with an 8-byte repeating pattern instead of a single byte like memset. 4006- PCI 4007 - Added support for using DPDK for PCI device mapping in addition to the 4008 existing libpciaccess option. Using the DPDK PCI support also allows use of 4009 the Linux VFIO driver model, which means that SPDK userspace drivers will work 4010 with the IOMMU enabled. Additionally, SPDK applications may be run as an 4011 unprivileged user with access restricted to a specific set of PCIe devices. 4012 - The PCI library API was made more generic to abstract away differences 4013 between the underlying PCI access implementations. 4014 4015## v1.2.0: IOAT user-space driver 4016 4017This release adds a user-space driver with support for the Intel I/O Acceleration Technology 4018(I/OAT, also known as "Crystal Beach") DMA offload engine. 4019 4020- IOAT 4021 - New user-space driver supporting DMA memory copy offload 4022 - Example programs `ioat/perf` and `ioat/verify` 4023 - Kernel-mode DMA engine test driver `kperf` for performance comparison 4024- NVMe 4025 - Per-I/O flags for Force Unit Access (FUA) and Limited Retry 4026 - Public API for retrieving log pages 4027 - Reservation register/acquire/release/report command support 4028 - Scattered payload support - an alternate API to provide I/O buffers via a sequence of callbacks 4029 - Declarations and `nvme/identify` support for Intel SSD DC P3700 series vendor-specific log pages and features 4030- Updated to support DPDK 2.2.0 4031 4032## v1.0.0: NVMe user-space driver 4033 4034This is the initial open source release of the Storage Performance Development Kit (SPDK). 4035 4036Features: 4037 4038- NVMe user-space driver 4039- NVMe example programs 4040 - `examples/nvme/perf` tests performance (IOPS) using the NVMe user-space driver 4041 - `examples/nvme/identify` displays NVMe controller information in a human-readable format 4042- Linux and FreeBSD support 4043