1# Changelog 2 3## v24.01: (Upcoming Release) 4 5### vhost 6 7Added `caw_iov` field to struct `spdk_scsi_task` to support SBC-3 compare_and_write IO. 8 9## v23.09 10 11### accel 12 13Added `spdk_accel_get_buf_align()` API to get minimum buffer alignment to execute a given operation. 14 15Added `get_operation_info()` callback for accel_modules to return constraints for a given operation. 16 17### bdev 18 19Added `spdk_bdev_open_async()` API to allow asynchronous bdev open that waits for specified time 20until the block device appears. 21 22### bdev_nvme 23 24Added `bdev_nvme_enable_controller` and `bdev_nvme_disable_controller` RPC to enable and disable 25NVMe controller. For NVMe multipath `cntlid` is used to select only one NVMe-oF controller. 26 27Added `cntlid` field to `bdev_nvme_reset_controller` RPC to support resetting only one NVMe-oF 28controller in multipath configuration. 29 30### bdev_raid 31 32Added `bdev_raid_remove_base_bdev` RPC to remove a base bdev from existing raid bdev. 33 34### dpdk 35 36Updated DPDK submodule to DPDK 23.07. 37 38### env 39 40The `phys_addr` parameter in `spdk_*_malloc()` functions is now invalid. Passing non-NULL value 41will return NULL from the functions. The parameter was deprecated in SPDK 19.04. 42For retrieving physical addresses, `spdk_vtophys()` should be used instead. 43 44### go 45 46Added JSON-RPC 2.0 client written in Go to ease communication with SPDK. 47Please see the documentation at `go/rpc/` and example at `examples/go/`. 48 49### init 50 51Options for the JSON-RPC server initialization were added. The options are defined via the 52`spdk_rpc_opts` structure and is passed to the existing API `spdk_rpc_initialize()` as a new 53argument. The options include `log_file` and `log_level`. 54 55### intel-ipsec-mb 56 57Updated intel-ipsec-mb submodule to v1.4. 58 59### jsonrpc 60 61New APIs, `spdk_jsonrpc_set_log_level` and `spdk_jsonrpc_set_log_file`, were added to enable 62logging JSON RPC calls history. 63 64### log 65 66New APIs, `spdk_flog` and `spdk_vflog`, were added to write messages to the specified log file. 67 68### nvme 69 70The `spdk_nvme_accel_fn_table` interface has been extended with callbacks allowing users to chain 71multiple accel operations. Users can now send requests with an existing accel sequence by setting 72the `accel_sequence` field in `spdk_nvme_ns_cmd_ext_io_opts` if the controller supports it (i.e. 73sets the `SPDK_NVME_CTRLR_ACCEL_SEQUENCE_SUPPORTED` flag) and the user implements the necessary 74`spdk_nvme_accel_fn_table` callbacks. 75 76The NVMe bdev will now advertise support for accel sequences if the `allow_accel_sequence` flag is 77set in `bdev_nvme_set_options` and the underlying controller also supports them. 78 79The TCP transport will now calculate data digest using the accel sequence APIs if they're available. 80 81Added `spdk_nvme_qpair_is_connected()` API to check the connection status. 82 83### nvmf 84 85The `spdk_nvmf_request::data` field has been removed: instead, clients should set 86`->iov` and `->iovcnt` appropriately, as nvmf request APIs now expect any data 87buffers to be described there. `spdk_nvmf_request_get_data()` has been removed. 88 89`transport` field in `listen_addresses` of `nvmf_get_subsystems` RPC is deprecated. 90`trtype` field should be used instead. `transport` field will be removed in 24.01 release. 91 92Added `spdk_nvmf_subsystem_add_listener_ext()` API for accepting new connections on the provided 93address, now allowing to specify additional options for the listener. 94 95Deprecated `spdk_nvmf_subsytem_any_listener_allowed()` API (to be removed in 24.01) and replaced with 96spdk_nvmf_subsystem_any_listener_allowed()` API fixing the typo. 97 98Added `spdk_nvmf_subsystem_is_discovery()` API to check whether a given susbystem is discovery subsystem. 99 100### scripts 101 102`setup.sh` now supports interactive mode for device selection and hugepage reservation. 103 104Added `backport.sh` script that formalizes process of backporting commits from latest SPDK, 105to specific release branches. Please see [documentation](https://spdk.io/doc/backporting.html). 106 107### thread 108 109Added `spdk_thread_is_app_thread()` API to check if specified spdk_thread is the app thread. 110 111### util 112 113Extended DIF support to CRC-32 and CRC-64 format of the Protection Information. 114 115Added `spdk_crc64_nvme()` and `spdk_crc32c_nvme()`, CRC-64 and CRC-32C checksums compatible with 116NVMe Protection Information. 117 118## v23.05 119 120### accel 121 122Added API `spdk_accel_submit_xor` to perform XOR. 123 124Configuration of `mlx5_pci` driver of `accel_dpdk_cryptodev` module was changed. Now key and key2 125should have equal length and key size in hexlified form can be either 32 bytes for 128b XTS key or 12664 bytes for 256b XTS key. Support of the wrapped crypto credentials mode for this driver is 127dropped, only plain text mode is supported. 128 129The accel library will now collect statistics on the number of executed operations and processed 130bytes. These statistics can be retreived using the `accel_get_stats` RPC. 131 132Added support for appending a crc32 calculation to an accel sequence. 133 134### bdev 135 136A new API `spdk_bdev_module_claim_bdev_desc` was added. Unlike `spdk_bdev_module_claim_bdev`, this 137function requires a bdev descriptor to be passed and the claim is automatically released when the 138descriptor is closed. It allows bdev modules to claim bdevs as a single writer, multiple writers, or 139multiple readers. 140 141New APIs `spdk_bdev_quiesce`, `spdk_bdev_unquiesce`, `spdk_bdev_quiesce_range` and 142`spdk_bdev_unquiesce_range` were added. These allow I/O to be quiesced on an entire bdev or 143a specified LBA range. 144 145Removed `spdk_bdev_ext_io_opts` from `spdk_bdev_io` and replaced it with fields that were missing in 146`spdk_bdev_io`. 147 148It is now possible to submit an I/O with an associated accel sequence. All operations in such 149sequence will be executed prior to sending the I/O (writes) or after its completion (reads). 150 151### env 152 153New functions `spdk_env_get_main_core` and `spdk_env_get_cpuset` were added. 154 155It is now possible to run applications on cores with IDs greater than 128 by passing a core map 156using the `--lcores` parameter. 157 158### examples 159 160`examples/nvme/perf` application now accepts `--use-every-core` parameter that changes 161the existing worker and namespace association logic to access every namespace from each worker. 162This replicates behavior of bdevperf application when `-C` option is provided. 163 164### gpt 165 166GPT bdevs now use the GPT Unique Partition ID as the bdev's UUID. 167 168### idxd 169 170Removed the `SPDK_IDXD_FLAG_PERSISTENT` flag. 171 172### lvol 173 174New API `spdk_lvol_iter_immediate_clones` was added to iterate the clones of an lvol. 175 176New APIs `spdk_lvol_get_by_uuid` and `spdk_lvol_get_by_names` to get lvols by the lvol's UUID or 177lvstore and lvol names. 178 179New `bdev_lvol_get_lvols` RPC to list logical volumes. This provides information about logical 180volumes without providing information about the bdevs. It is useful for listing the lvols 181associated with specific lvol stores and for listing lvols that are degraded and have no 182associated bdev. 183 184Added support for external snapshots making it possible to create clones of any (including non-lvol) 185bdev. Users can create such clones using the `bdev_lvol_clone_bdev` RPC. 186 187### nvme 188 189New API `spdk_nvme_ns_get_format_index` was added to calculate the exact format index, that 190was used to format the namespace. 191 192Added two new APIs `spdk_nvme_ns_cmd_io_mgmt_recv` and `spdk_nvme_ns_cmd_io_mgmt_send` to 193receive and send the I/O management commands. 194 195### nvmf 196 197New `spdk_nvmf_request_copy_to/from_buf()` APIs have been added, which support 198iovecs, unlike the deprecated `spdk_nvmf_request_get_data()`. 199 200Two functions related to Asynchronous Event and error handling have been made public: 201 202- `spdk_nvmf_ctrlr_async_event_error_event`, 203- `spdk_nvmf_ctrlr_abort_aer`. 204 205New `spdk_nvmf_transport_create_async` was added, it accepts a callback and callback argument. 206`spdk_nvmf_transport_create` is marked deprecated. 207 208Parameters `cb_fn` and `ctx` of `spdk_nvmf_qpair_disconnect` API are deprecated. These parameters 209will be removed in 23.09 release. 210 211Added a `secure_channel` parameter to the `nvmf_subsystem_add_listener` RPC. When true, all 212connections established via this listener will immediately attempt to establish a secure channel, 213prior to any authentication. Only valid for the TCP transport. 214 215Added two optional transport module callbacks: `subsystem_add_host()` and `subsystem_remove_host()`. 216These functions will notify a transport about adding/removing hosts' access. 217 218TLS PSK identity is now generated from subsystem NQN and host NQN. PSK interchange format is now 219expected as input, when configuring TLS in SPDK. TLS feature is considered experimental. 220 221### part 222 223New API `spdk_bdev_part_construct_ext` is added and allows the bdev's UUID to be specified. 224 225### sock 226 227Added a zero copy receive interface allowing users to provide buffers to a socket group (via 228`spdk_sock_group_provide_buf()`), which will then get filled with the next portions of the stream on 229each socket. After that, pointer to the buffer holding the next part of the data can be obtained 230using `spdk_sock_recv_next()`. 231 232Added a callback `get_key()` to `spdk_sock_impl_opts` structure. 233 234New function `spdk_sock_get_impl_name()` was added to retrieve current socket implementation name. 235 236### thread 237 238Added two new APIs `spdk_thread_bind` and `spdk_thread_is_bound` to bind or unbind `spdk_thread` 239to its current CPU core, and check bound status. 240 241### ublk 242 243Enabled support for `UBLK_F_NEED_GET_DATA`. The buffers are no longer pre-allocated during 244initialization, but are allocated on-demand. Additionally, they're now allocated from the `iobuf` 245buffer pools. 246 247### util 248 249New APIs `spdk_uuid_is_null` and `spdk_uuid_set_null` were added to compare and 250set UUID to NULL value. 251 252Added new functions, `spdk_fd_group_{nest,unnest}`, providing efficient way of adding one 253`fd_group`'s file descriptors to another `fd_group`. 254 255## v23.01 256 257### accel 258 259New library isa-l-crypto has been added, it is used by accel library in crypto operations. 260 261New functions `spdk_accel_submit_encrypt` and `spdk_accel_submit_decrypt` were added. 262 263New accel module `dpdk_cryptodev` has been added. It uses DPDK crypto PMD and support encrypt and 264decrypt operations. New RPC `dpdk_cryptodev_scan_accel_module` has been added to enable this accel module. 265 266New accel module `mlx5` was added. It implements crypto operations, enabled when SPDK is configured with 267RDMA provider is mlx5_dv and crypto support. 268 269Introduced the concept of chaining multiple accel operations together and executing them all in a 270single call. The operation can be chained via one of the `spdk_accel_append_*` functions and then 271executed using `spdk_accel_sequence_finish`. Currently, copy, decompress, decrypt, encrypt, and 272fill operations support this mode. This feature is considered experimental. 273 274### bdev 275 276A new API `spkd_bdev_part_submit_request_ext` was added to specify a custom completion callback. 277 278New APIs `spdk_bdev_for_each_channel` and `spdk_bdev_for_each_channel_continue` and 279associated function pointers were added to iterate each channel of the required bdev. 280 281The RPC `bdev_get_iostat` now allows a user to query the per channel IO statistics for 282required bdev, and displays maximum and minimum latencies and I/O error counts. 283 284New `spdk_bdev_copy_blocks` and `spdk_bdev_get_max_copy` APIs to support copy commands. 285 286A new API `spdk_bdev_io_get_submit_tsc` was added to get the submit_tsc of the bdev I/O. 287 288Bdevs will no longer have UUIDs generated based on timestamp and are responsible for 289setting this field themselves. Generation of UUIDs for NVMe bdevs may be enabled by 290running `bdev_nvme_set_options` RPC with `--generate-uuids` option. These identifiers 291are based on serial number and namespace ID and will always be the same for a given 292device. 293 294A new API `spdk_bdev_channel_get_histogram` was added to get the histogram of a specified 295channel for a bdev. 296 297Converted internal use of `pthread_mutex_t` to `struct spdk_spinlock`. Consumers of bdev 298API functions must be on an SPDK thread or the program will abort. It is now enforced 299that no internal bdev locks can be held when a poller or message goes off CPU. 300 301A new RPC `bdev_reset_iostat` was added to reset I/O statistics of bdevs. Note that if one 302consumer reset I/O statistics, it affects all other consumers. 303 304Add function pointers, `dump_device_stat_json` and `reset_device_stat` to the bdev module 305function table to display and reset I/O statistics specific for the module specific bdev 306context. 307 308New APIs `spdk_bdev_reset_io_stat`, `spdk_bdev_add_io_stat` and `spdk_bdev_dump_io_stat_json` 309were added to process I/O statistics outside the generic bdev layer, especially in bdev modules. 310 311### bdev_crypto 312 313vbdev_crypto is updated to use accel framework instead of DPDK PMDs. 314 315### bdev_daos 316 317New RPC `bdev_daos_resize` was added to allow resizing the daos bdev. 318 319### bdev_malloc 320 321Both of interleaved and separated metadata are now supported by the malloc bdev module. 322 323Protection information is now supported by the malloc bdev module. 324 325### bdev_nvme 326 327Updated `bdev_nvme_set_options` RPC (and rpc.py) to support the new `transport_tos` parameter. 328 329For the active-active policy of the multipath mode, in addition to the default round-robin path 330selector, the minimum queue depth path selector was added. The minimum queue depth path selector 331selects an I/O path according to the number of outstanding requests of each nvme qpair. 332 333Added RPCs `bdev_nvme_start_mdns_discovery`, `bdev_nvme_get_mdns_discovery_info` and 334`bdev_nvme_stop_mdns_discovery` to perform Avahi based mDNS discovery service, 335as per NVMe TP 8009 - Automated Discovery of Ethernet Discovery Controllers. 336 337A new option `nvme_error_stat` was added to the `bdev_nvme_set_options` RPC to enable 338collecting NVMe error counts. 339 340Added I/O statistics per I/O path to the NVMe bdev module for NVMe bdev multipath. It can be 341enabled by a new option `io_path_stat` of RPC `bdev_nvme_set_options`. 342 343Added RPC `bdev_nvme_get_path_iostat` to get I/O statistics for IO paths of the NVMe bdev. 344 345Added `selector` parameter to `bdev_nvme_set_multipath_policy` RPC to set path selector for multipath. 346Option `round_robin` and `queue_depth` are available. 347 348Added `rr_min_io` option to RPC `bdev_nvme_set_multipath_policy`. It switches I/O to 349another path after rr_min_io I/Os are routed to current io path for the round-robin 350path selector. 351 352Added option `--io-path-stat` for RPC `bdev_nvme_set_option` to enable collecting io path stat. 353 354### bdev_raid 355 356Changed `bdev_raid_get_bdevs` RPC output format to include raid_bdev details. 357 358### bdevperf 359 360Promoted the application to example to match similar programs: fio_plugin and perf. 361It can now be found inside `examples/bdev/bdevperf`. 362 363### env 364 365Added `spdk_mempool_mem_iter` that allows to get start address and length of each memory 366chunk in order to create app-specific resources. 367 368### event 369 370Added core lock file mechanism to prevent the same CPU cores from being used by multiple 371SPDK application instances. 372 373Added `--disable-cpumask-locks` command line switch to disable CPU locks on SPDK startup. 374 375Added RPCs `framework_enable_cpumask_locks` and `framework_disable_cpumask_locks` to enable 376and disable CPU core locks in runtime. 377 378Added `--rpcs-allowed` command line option. Users can specify a comma-separated list of RPC 379names with this option to restrict allowed RPCs to only that list. 380 381Added `--vfio-vf-token` command line option to specify a VF token (UUID) 382shared between SR-IOV PF and VFs for vfio_pci driver. 383 384The `spdk_iov_xfer_*()` APIs were added for handling incremental copying between 385an iovec and a buffer. 386 387### json 388 389Added API `spdk_json_write_double` and `spdk_json_write_named_double` to allow 390for writing and decoding of the the double data type. 391 392### log 393 394Added consistent tracking of use of deprecated behavior. Deprecations continue to be 395listed in deprecation.md and now are in subsections with headers matching a tag 396used in `SPDK_LOG_DEPRECATED()` calls. When deprecated behavior is used, these 397tags will appear in SPDK's log at the warn level. As the SPDK application exits, it 398will log a summary of how many times `SPDK_LOG_DEPRECATED()` was called for each 399tag that was logged at least once. 400 401### nvme 402 403NVMe transport options were newly introduced. The NVMe transport options are defined via 404the `spdk_nvme_transport_opts` structure and configured via `spdk_nvme_transport_get_opts` 405and `spdk_nvme_transport_get_opts` functions. 406 407Shared receive queue was supported by the RDMA transport. It can be configured by 408a new NVMe transport option `rdma_srq_size`. 409 410Added `transport_tos` to `spdk_nvme_ctrlr_opts` to support setting of the "type of service" 411value in the IPv4 header. Only RDMA is supported at this time. 412 413Added API `spdk_nvme_qpair_get_num_outstanding_reqs` to get the number of outstanding reqs of 414a specified qpair. 415 416### nvmf 417 418Added API `spdk_nvmf_tgt_pause_polling` and `spdk_nvmf_tgt_resume_polling` to allow 419pausing polling on poll group of a given target. 420 421The `num_shared_buffers` option now controls the maximum number of buffers a transport will 422pull from the central `iobuf` pool. It will not allocate additional memory. If the requested 423amount cannot be satisfied, a warning will be displayed to increase the size of the `iobuf` 424pool. 425 426### rpc 427 428Added `spdk_rpc_set_allowlist` to restrict allowed RPCs to the specified list. 429 430### scheduler 431 432Changing scheduler from dynamic back to static is no longer possible, 433as there is no way of saving original SPDK thread distribution on reactors. 434 435### thread 436 437Added `spdk_thread_get_app_thread` which returns the first thread that was created using 438`spdk_thread_create`. 439 440Added `spdk_thread_is_running`. This returns `true` for a running thread, or `false` if 441its exit process has started using `spdk_thread_exit`. 442 443Added API `spdk_spin_init`, `spdk_spin_destroy`, `spdk_spin_lock`, `spdk_spin_unlock`, and 444`spdk_spin_held` to support spinlocks that are aware of the SPDK concurrency model. 445 446Added iobuf buffer pool, set of API calls for allocating data buffers across libraries. 447Please see new API `spdk_iobuf_*`. 448 449### trace 450 451New `trace_get_info` RPC was added to get name of shared memory file, list of the 452available trace point groups and mask of the available trace points for each group. 453 454### ublk device 455 456The ublk application supports the ublk kernel driver. It's implemented as a ublk backend 457in spdk_tgt and could be started with specifying configuration. See the 458[ublk](https://www.kernel.org/doc/html/latest/block/ublk.html) documentation for more details. 459 460ublk bdev could export a block device via Linux ublk. It will move this backend device into userspace 461as `/dev/ublkb*`. Before to adding ublk device, need to create ublk target by RPC method. 462 463### util 464 465New API `spdk_fd_group_get_epoll_event` that returns the epoll(7) event that 466caused a function callback in file descriptor group to execute. 467 468Added API `spdk_strarray_from_string`, `spdk_strarray_dup` and `spdk_strarray_free` 469for handling arrays of strings. 470 471A new API `spdk_strcpy_replace` was added to replace all occurrences of the search string 472with the replacement string. 473 474New API `spdk_iov_memset()` was added to memset an iovec. 475 476New API `spdk_iov_one()` was added to initialize an iovec for a single buffer. 477 478## v22.09 479 480### accel 481 482Many names were changed in the accel framework to make them consistent both with themselves and 483the rest of SPDK. The primary public header file is now named `include/spdk/accel.h`. 484 485Added a new runtime RPC `accel_get_opc_assignments` to get a list of current opcode to engine 486assignments. 487 488Added a new startup RPC `accel_assign_opc` to assign/override a specific opcode to 489an engine. 490 491### bdev 492 493New RPCs `bdev_xnvme_create` and `bdev_xnvme_delete` were added to support the xNVMe bdev. 494 495A new API `spdk_bdev_for_each_bdev_io` was added to execute the function on the appropriate 496thread for each bdev_io submitted to the bdev. 497 498A new API `spdk_bdev_get_current_qd` was added to measure and return the queue depth from a 499bdev. This API is available even when queue depth sampling is disabled. 500 501Added `spdk_bdev_seek_data`, `spdk_bdev_seek_hole` and `spdk_bdev_io_get_seek_offset` functions 502that start from a given offset and seek for next data or for next hole. At this time only 503supported by lvol bdev module. 504 505A new parameter `io_drain_timeout` has been added to `spdk_bdev` structure. It controls how long 506a bdev reset must wait for IO to complete prior to issuing a reset to the underlying device. 507If there is no outstanding IO at the end of that period, the reset is skipped. 508Best defined for bdevs that share an underlying bdev, such as multiple lvol bdevs sharing 509an nvme device, to avoid unnecessarily resetting the underlying bdev and affecting other 510bdevs that are sharing it. 511Modifying this field is optional. Setting the value to 0 keeps the original behavior 512to always send resets immediately, even if there is no I/O outstanding. 513 514### blobstore 515 516Reserve space for `used_clusters` bitmap. The reserved space could be used for blobstore growing 517in the future. 518 519Added `is_zeroes` operation to `spdk_bs_dev`. It allows to detect if logical blocks are backed 520by zeroes device and do a shortcut in copy-on-write flow by excluding copy part from zeroes device. 521 522Added `spdk_blob_get_next_allocated_io_unit` and `spdk_blob_get_next_unallocated_io_unit` functions 523that start from a given offset and seek for first io_unit belonging to an allocated cluster 524or first io_unit belonging to an unallocated cluster. 525 526Added `spdk_bs_grow` function to grow blobstore size if the underlying bdev size is increased. 527This is used by lvol bdev via `bdev_lvol_grow_lvstore` RPC. 528 529### daos 530 531Added new DAOS bdev module, that creates SPDK block device on top of DAOS DFS. 532Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_daos) for more details. 533 534### env 535 536Added `spdk_pci_register_device_provider` and matching `SPDK_PCI_REGISTER_DEVICE_PROVIDER` 537functions for registering PCI device providers. That allow the VMD driver to be notified 538when users want to attach a device under a given BDF and when a device is detached. 539 540### init 541 542`spdk_subsystem_init_from_json_config` now fails if the JSON configuration file is not 543an object with an array named "subsystems". 544 545### iscsi 546 547Added `bdev_iscsi_set_options` RPC to modify options of the iSCSI bdev module. 548 549### json 550 551Added `spdk_json_find` API return errcode: EPROTOTYPE - json not enclosed in {}. 552`spdk_json_find` now returns -EPROTOTYPE instead of -ENOENT if the object parameter 553does not point to a JSON object (i.e. is not enclosed with {}). 554 555### lvol 556 557Add `num_md_pages_per_cluster_ratio` parameter to the `bdev_lvol_create_lvstore` RPC. 558Calculate `num_md_pages` from `num_md_pages_per_cluster_ratio`, and pass it to `spdk_bs_opts`. 559 560### nvme 561 562Added SPDK_NVME_TRANSPORT_CUSTOM_FABRICS to enum `spdk_nvme_transport_type` to support custom 563fabric transport. SPDK_NVME_TRANSPORT_CUSTOM was intended to be non-fabric custom transport. 564 565Added a new function `spdk_nvme_ns_cmd_verify` to submit a Verify Command to a Namespace. 566 567Added `spdk_nvme_ctrlr_disable_read_changed_ns_list_log_page` to allow an application to 568tell the driver to not read the CHANGED_NS_LIST log page in response to a NS_ATTR_CHANGED 569AEN. When called the application is required to read this log page instead to clear the 570AEN. 571 572Added `psk` field to `spdk_nvme_ctrlr_opts` struct in order to enable SSL socket implementation 573of TCP connection and set the PSK. Applicable for TCP transport only. 574 575### raid 576 577Renamed the `raid5` module to `raid5f` to reflect that it is not a traditional 578RAID5 implementation - only full stripe writes are supported, partial stripe 579writes (read-modify-write) are not. 580 581### rpc 582 583Added `psk` parameter to `bdev_nvme_attach_controller` RPC in order to enable SSL socket implementation 584of TCP connection and set the PSK. Applicable for TCP transport only. 585 586New options `enable_ktls` and `tls_version` were added to the `sock_impl_set_options` structure. 587New options `psk_key` and `psk_identity` were added to the `sock_impl_set_options` structure. 588 589Added warning message for `bdev_rbd_create`, if it is used without -c. 590`bdev_rbd_create()` API without specifying -c is deprecated and will be removed in future release. 591 592Renamed `enable_vmd` RPC to `vmd_enable` to make it consistent with our naming scheme of 593`<subsystem>_<action>`. For now, the old name is still available, but is marked as deprecated. 594 595New function `spdk_rpc_get_method_state_mask` was added to RPC library. 596 597### sma 598 599Extended `VolumeParameters` with crypto parameters allowing the user to configure crypto when 600attaching a volume to a device. This interface is now supported by all upstream device types 601(nvmf-tcp, virtio-blk, vfio-user) using `bdev_crypto`. Users must specify the crypto engine to use 602under `crypto` section in config. It is also possible to register out-of-tree crypto engines by 603inheriting from the `CryptoEngine` class. 604 605Added two new methods: `SetQos` and `GetQosCapabilities` allowing the user to configure QoS on a 606per-device or per-volume level. Not all QoS settings have to be supported by each device, so users 607can use `GetQosCapabilities` to query them for that. All upstream device types support QoS on a 608per-volume level using bdev layer's QoS mechanism. 609 610### sock 611 612Added new `ssl` based socket implementation, the code is located in `module/sock/posix`. 613For now we are using hard-coded PSK and only support TLS 1.3. 614 615### util 616 617Added new functions: `spdk_hexlify` and `spdk_unhexlify`. 618 619A new API `spdk_xor_gen` was added to generate XOR from multiple source buffers. It is going to be 620used by `raid5f` for calculating parity. 621 622### vfu_tgt 623 624Added vfu_tgt library abstraction based on libvfio-user for PCI device emulation. 625Besides the NVMe device emulation, its now possible to emulate virtio-blk and virtio-scsi 626devices. 627 628### virtio 629 630virtio-vhost-user no longer tries to support dynamic memory allocation. The vhost target does 631not support the high rate of SET_MEM_TABLE vhost messages that result from dynamic memory 632allocation, so a virtio-vhost-user device will now present an ERRLOG, assert, and skip the 633SET_MEM_TABLE vhost message if a memory notification is received outside of the normal device 634start/stop. Applications using the virtio library in vhost-user mode should now pre-allocate 635the application's memory using the -s/--mem-size option and use single shared memory file 636segments using the -g/--single-file-segments option. 637 638Added `vfio-user` transport type for virtio-blk and virtio-scsi devices. 639 640### vmd 641 642Fixed hotplug when a device is inserted in a slot in which a disk was already enumerated previously. 643Added two new RPCs: `vmd_remove_device` simulating a hotremove, and `vmd_rescan`, which rescans all 644buses managed by the VMD driver and hotplugs all newfound devices. 645 646## v22.05 647 648### sock 649 650A new option `ack_timeout` was added to the `spdk_sock_opts` structure. 651 652### util 653 654A new parameter `bounce_iovcnt` was added to `spdk_dif_generate_copy` and `spdk_dif_verify_copy`. 655The `bounce_iovcnt` is used to specify the number of bounce_iov to support multiple block-aligned 656fragment copies. 657 658A new API `spdk_copy_iovs_to_buf` and `spdk_copy_buf_to_iovs` were added to copy iovs to buf or 659copy buf to iovs. There're many cases need to use these two APIs. 660 661### bdev 662 663Removed deprecated spdk_bdev_module_finish_done(). Use spdk_bdev_module_fini_done() instead. 664 665A new API `spdk_bdev_unregister_by_name` was added to handle race conditions correctly. 666 667New APIs, `spdk_for_each_bdev` and `spdk_for_each_bdev_leaf`, were added to provide iteration 668safe for race conditions. 669 670A new RPC `bdev_nvme_get_io_paths` was added to get all active I/O paths. 671 672A new RPC `bdev_nvme_set_preferred_path` was added to set preferred I/O path for an NVMe bdev 673when in multipath mode. This RPC does not support NVMe bdevs in failover mode. 674 675A new RPC `bdev_nvme_set_multipath_policy` was added to set multipath policy of a NVMe bdev 676in multipath mode. 677 678A new option `disable_auto_failback` was added to the `bdev_nvme_set_options` RPC to disable 679automatic failback. 680 681### idxd / dsa 682 683A new parameter `flags` was added to all low level submission and preparation 684APIs to enable the caller to pass operation flags per the DSA specification. 685 686A new flag 'SPDK_IDXD_FLAG_PERSISTENT' was added to let DSA know that 687the destination is persistent. 688 689The RPC `idxd_scan_accel_engine` has been renamed to `dsa_scan_accel_engine` 690 691The RPC `iaa_scan_accel_engine` has been added. 692 693Many HW related structs/functions with the name `idxd` have been renamed `dsa` 694to more accurately represent the HW they are associated with. 695 696Two new functions were added to the library `spdk_idxd_submit_compress` and 697`spdk_idxd_submit_decompress` 698 699### accel_fw 700 701A new parameter `flags` was added to accel API. 702The APIs include: 703`spdk_accel_submit_copy` 704`spdk_accel_submit_dualcast` 705`spdk_accel_submit_fill` 706`spdk_accel_submit_copy_crc32c` 707`spdk_accel_submit_copy_crc32cv` 708`spdk_accel_submit_compress` 709`spdk_accel_submit_decompress` 710 711A new flag `ACCEL_FLAG_PERSISTENT` was added to indicate the target memory is PMEM. 712 713The API `spdk_accel_get_capabilities` has been removed. 714 715### crypto 716 717Support for AES_XTS was added for MLX5 polled mode driver (pmd). 718 719bdev_crypto_create RPC now requires hexlified 'key' and 'key2' params for all pmd drivers. 720Unhexlifying is performed during RPC command processing and the vbdev crypto module runs on 721binary keys as before. 722 723### bdev 724 725Added a timeout option to the `bdev_get_bdevs` RPC. It allows the user to specify the amount of 726time to wait until a bdev with a given name appears in the system. 727 728### bdev_nvme 729 730Added `bdev_nvme_add_error_injection` and `bdev_nvme_remove_error_injection` RPCs to add and 731remove NVMe error injections. 732 733New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 734added to the RPC `bdev_nvme_set_options`. They can be overridden if they are given by the RPC 735`bdev_nvme_attach_controller`. 736 737### blobstore 738 739New functions `spdk_blob_io_writev_ext` and `spdk_blob_io_readv_ext` are added. The new functions accept 740`spdk_blob_ext_io_opts` structure with extended IO request options. 741 742### event 743 744Added `msg_mempool_size` parameter to `spdk_reactors_init` and `spdk_thread_lib_init_ext`. 745The size of `g_spdk_msg_mempool` can now be controlled through the same-named 746user option of `spdk_app_opts` structure. 747 748### nvme 749 750The API `spdk_nvme_ctrlr_prepare_for_reset()` was deprecated. The functionality provided by the 751`spdk_nvme_ctrlr_prepare_for_reset()` was merged into the API `spdk_nvme_ctrlr_disconnect()`. 752 753### nvmf 754 755Removed deprecated max_qpairs_per_ctrlr parameter from nvmf_create_transport RPC. Use 756max_io_qpairs_per_ctrlr instead. 757 758Deprecated the ability for hosts to connect to the discovery subsystem automatically on any 759existing listener. Users should now explicitly add listeners for the discovery subsystem. 760Host can still connect to the discovery subsystem as before, but a warning message will be 761emitted if no listener was configured for the transport ID of the incoming connection. 762 763Added adaptive interrupt feature for vfio-user transport. New parameter `disable_adaptive_irq` 764is added to the RPC `nvmf_create_transport`. 765 766### thread 767 768Added `spdk_thread_exec_msg()` API. 769 770### scheduler 771 772`framework_set_scheduler` can now be called after application initialization. 773Added callbacks to set custom parameters specific for each scheduler implementation 774and `framework_get_scheduler` to retrieve them. 775 776Added `dynamic` scheduler options: load_limit, core_limit, core_busy. Their descriptions 777are available in JSON-RPC document, in section 778[framework_set_scheduler](jsonrpc.html#rpc_framework_set_scheduler). 779 780### raid 781 782Add concat as a special raid module. The concat module could create a virtual bdev. The 783virtual bdev combines multiple underlying bdevs together. The layout of the underlying 784bdevs is one after another. The concat bdev is extendable. When the free space of the 785concat bdev is not enough, the user can deconstruct the concat bdev, then reconstruct it 786with an additional underlying bdev. 787 788### sock 789 790Allow MSG_ZEROCOPY flag to be set or not according to data size, which can be enabled and 791set by setting "zerocopy_threshold". zerocopy_threshold = 0 means disable this function; 792zerocopy_threshold > 0 means enable it and use this value as the threshold. 793 794### rpc 795 796Introduced `zerocopy_threshold` to enable zerocopy on send for server sockets according to 797data size to be flushed. 798 799## v22.01 800 801### accel 802 803The batching capability was removed. Batching is now considered an implementation 804detail of the low level drivers. 805 806### bdev 807 808The NVMe bdev module supports multipath and improved I/O error resiliency. 809 810The parameter `retry_count` of the RPC `bdev_nvme_set_options` was deprecated and will be 811removed in SPDK 22.04, and the parameter `transport_retry_count` is added and used instead. 812 813An new parameter `bdev_retry_count` is added to the RPC `bdev_nvme_set_options`. 814 815New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 816added to the RPC `bdev_nvme_attach_controller`. 817 818An new parameter `num_io_queues` is added to `bdev_nvme_attach_controller` RPC to allow specifying amount 819of requested IO queues. 820 821Added `key_file` parameter to the `rbd_register_cluster` RPC. It is an optional parameter to 822specify a keyring file to connect to a RADOS cluster. 823 824Added `spdk_bdev_get_zone_id` API to retrieve zone_id for a certain LBA. 825 826### bdev_aio 827 828Added `bdev_aio_rescan` RPC to allow rescanning the size of aio bdev. 829 830### bdev_nvme 831 832Added discovery service to bdev_nvme module, which now can connect to a discovery controller. 833bdev_nvme will connect to all current and future subsystems in the discovery controller. 834Added `bdev_nvme_start_discovery` and `bdev_nvme_stop_discovery` RPC. 835 836### dpdk 837 838Updated DPDK submodule to DPDK 21.11. 839 840### env 841 842Added `spdk_pci_for_each_device`. 843 844Removed `spdk_pci_get_first_device` and `spdk_pci_get_next_device`. These APIs were unsafe, because 845they did not account for PCI devices being inserted or removed while the caller was using handles 846returned from these APIs. Existing users of these APIs should switch to `spdk_pci_for_each_device` 847instead. 848 849Added 3 experimental APIs to handle PCI device interrupts (`spdk_pci_device_enable_interrupt`, 850`spdk_pci_device_disable_interrupt`, `spdk_pci_device_get_interrupt_efd`). 851 852Added `framework_get_pci_devices` RPC to list PCIe devices attached to an SPDK application. 853 854### idxd 855 856Many APIs are now vectored rather than scalar, meaning they take iovecs instead of individual pointers. 857 858### json 859 860Added `spdk_json_write_bytearray` API to serialize a buffer as a hex string. 861 862### nvme 863 864API `spdk_nvme_trtype_is_fabrics` was added to return existing transport type 865is fabric or not. 866 867API `spdk_nvme_poll_group_remove` was limited to be available only for a 868disconnected qpair in the group. 869 870New APIs, `spdk_nvme_ctrlr_disconnect`, `spdk_nvme_ctrlr_reconnect_async`, and 871`spdk_nvme_ctrlr_reconnect_poll_async`, have been added to improve error recovery, and 872the existing APIs,`spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async` 873were deprecated. 874 875Added `spdk_nvme_ctrlr_get_discovery_log_page` API for getting the full discovery log page 876from a discovery controller. 877 878### nvmf 879 880Added support for zero-copy operations in the NVMe-oF TCP target. It can be enabled via 881the `zcopy` parameter when creating a transport. The zero-copy operations are only used 882for requests not using in-capsule data. 883 884Added a `subsystem` parameter to `spdk_nvmf_transport_stop_listen_async`. When not NULL, 885it will only disconnect qpairs for controllers associated with the specified subsystem. 886 887Removed accept poller from transport layer. Each transport can have its own policy of 888handling new connections. To notify transport layer `spdk_nvmf_poll_group_add` and 889`spdk_nvmf_tgt_new_qpair` can be used. 890 891### scsi 892 893Structure `spdk_scsi_lun` has been extended with new member `resizing` so that SCSI layer now reports 894unit attention for disk resize. 895 896### trace 897 898Added `spdk_trace_create_tpoint_group_mask` to return tracepoint group mask from 899a tracepoint group name. 900 901Added `trace_set_tpoint_mask` and `trace_clear_tpoint_mask` RPC to allow enabling 902individual traces. 903 904### util 905 906Added `spdk_ioviter_first` and `spdk_ioviter_next` to iterate over two iovecs and 907yield pointers to common length segments. 908 909### sock 910 911A new parameter `hint` is added to `spdk_sock_get_optimal_sock_group`. It allows to suggest 912a poll group when no optimal poll group is found. 913 914## v21.10 915 916Structure `spdk_nvmf_target_opts` has been extended with new member `discovery_filter` which allows to specify 917filtering rules applied during discovery log generation. Refer to `enum spdk_nvmf_tgt_discovery_filter` for more info. 918 919### bdev 920 921New API `spdk_bdev_get_memory_domains` has been added, it allows to get SPDK memory domains used by bdev. 922 923New API functions `spdk_bdev_readv_blocks_ext` and `spdk_bdev_writev_blocks_ext` have been added. 924These functions accept `spdk_bdev_ext_io_opts` structure with extended IO request 925options, e.g. DMA memory domain which describes data that may belong to another memory domain and 926can't be accessed directly. 927 928Added `async_fini_start` to allow bdev modules to complete the `fini_start` asynchronously, 929with new `spdk_bdev_module_fini_start_done` API. 930 931Deprecated `spdk_bdev_module_finish_done()` API, which will be removed in SPDK 22.01. 932Bdev modules should use `spdk_bdev_module_fini_done()` instead. 933 934The `ocssd` bdev has been removed. The Open Channel specification has been largely superseded by 935zoned namespaces, few if any devices have been brought to market, and there is little reason to 936continue to support this. OCSSD support in the nvme driver will remain for now. 937 938### dma 939 940A new library, lib/dma, has been added. This library provides the necessary infrastructure for 941handling systems and devices with multiple memory domains. For example, a PCIe add-in card with an 942SoC may be running SPDK on the SoC. That SoC has its own local memory, but SPDK may be controlling 943devices that can also access the host system memory. This library provides infrastructure to enumerate 944the memory domains and request hardware perform DMA transfers between them. 945 946### dpdk 947 948Updated DPDK submodule to DPDK 21.08. 949 950### event 951 952Added the `disable_signal_handlers` flag to the `spdk_app_opts` struct. 953 954### idxd 955 956Added `spdk_idxd_get_socket` to query the socket that the idxd device 957is on. 958 959### json 960 961Added API to allow for writing and decoding of new types: 962 963- `spdk_json_write_named_uint16` 964- `spdk_json_write_named_uint8` 965- `spdk_json_write_uint16` 966- `spdk_json_write_uint8` 967- `spdk_json_number_to_uint8` 968- `spdk_json_decode_uint8` 969 970### log 971 972Added API `spdk_log_to_syslog_level` to return syslog level based on SPDK's 973log level. 974 975### nvme 976 977Added new functions `spdk_nvme_zns_set_zone_desc_ext` and `spdk_nvme_zns_ext_report_zones` 978to set zone descriptor extension and to get extended zone report respectively. 979 980New API `spdk_nvme_ctrlr_get_memory_domains` has been added, it allows to get SPDK memory domains used by nvme controller. 981 982New API functions `spdk_nvme_ns_cmd_readv_ext` and `spdk_nvme_ns_cmd_writev_ext` 983have been added. These functions accept `spdk_nvme_ns_cmd_ext_io_opts` structure with extended IO request 984options, e.g. DMA memory domain which describes data that may belong to another memory domain and 985can't be accessed directly. 986 987Added a new function `spdk_nvme_ctrlr_get_regs_bpinfo` to get boot partition info of a controller. 988Added new functions `spdk_nvme_ctrlr_write_boot_partition`, 989`spdk_nvme_ctrlr_read_boot_partition_start` and `spdk_nvme_ctrlr_read_boot_partition_poll` 990to write and read the boot partitions of a controller. 991 992Added `spdk_nvme_ctrlr_get_opts` to retrieve the current controller options. 993 994Added `async_mode` to `spdk_nvme_io_qpair_opts` to enable creation of submission and completion 995queues asynchronously. This mode is currently supported at PCIe layer, 996which tracks the qpair creation with state machine and returns to the user immediately. 997Default mode is set to false to create io qpairs synchronously. 998 999Added `spdk_nvme_ctrlr_get_regs_cc` to retrieve NVMe controller CC (Configuration) register. 1000 1001Added `spdk_nvme_ctrlr_prepare_for_reset` to inform the driver that the application is preparing 1002to reset the specified NVMe controller. This function allows the driver to make decisions 1003knowing that a reset is about to happen. 1004 1005### nvmf 1006 1007Added `oncs` to `struct spdk_nvmf_ctrlr_data` so that the transport layer 1008can decide support RESERVATION feature or not. 1009 1010An `opts_size` element was added in the `spdk_nvmf_ns_opts` structure to solve the 1011ABI compatibility issue between different SPDK version. An new option `anagrpid` was 1012added in the `spdk_nvmf_ns_opts` structure. 1013 1014An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_add_ns`. 1015 1016An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_listener_set_ana_state`. 1017 1018`spdk_nvmf_subsystem_destroy` is now can be asynchronous, it accepts a callback and callback argument. 1019 1020A new parameter, `poll_groups_mask` was added to the `nvmf_set_config` RPC that allows specifying 1021a subset of cores for the nvmf poll groups. This helps to avoid imbalances when some cores are 1022busy with periodic timer tasks that run very frequently. 1023 1024A new parameter, `commit` was added to `nvmf_bdev_ctrlr_end_zcopy` that allows indicating whether 1025the buffers should be committed. 1026 1027Added `listen_dump_opts` to `spdk_nvmf_transport_ops` so that the transport layer can display 1028listen opts. 1029 1030Added `fabrics_connect_timeout_us` to `bdev_nvme_attach_controller` RPC to allow specifying 1031timeout for connect operation. 1032 1033Added `multipath` to `bdev_nvme_attach_controller` RPC to allow specifying multipath behavior. 1034 1035### scheduler 1036 1037New API for implementing schedulers and governors. Please see `include/spdk/scheduler.h` for details. 1038 1039### scsi 1040 1041New functions, `spdk_scsi_dev_get_first_lun` and `spdk_scsi_dev_get_next_lun` 1042have been added to iterate LUNs of a SCSI device. 1043 1044Each SCSI device supports 256 LUNs at the maximum now and the macro constant 1045`SPDK_SCSI_DEV_MAX_LUN` was removed. 1046 1047### trace_parser 1048 1049A new library, lib/trace_parser, has been added. This library provides functions that parse traces 1050recorded by an SPDK application. That includes merging traces from multiple cores, 1051sorting them by their timestamp and constructing trace entries spanning across multiple buffers. 1052 1053### util 1054 1055The `spdk_fd_group_add` API now takes a `name` parameter. 1056 1057## v21.07 1058 1059### accel_fw 1060 1061Added API `spdk_accel_submit_copy_crc32c` to perform a CRC32C while copying data. 1062 1063Added API `spdk_accel_batch_prep_copy_crc32c` to batch CRC32C + copy commands. 1064 1065Added API `spdk_accel_submit_copy_crc32cv` to submit chained CRC32C + copy commands. 1066 1067### bdev 1068 1069Change `spdk_bdev_read_blocks_with_md` arg offset definition from int64_t to uint64_t. 1070 1071Red-black tree has been used to organize the bdev names and aliases uniformly 1072to provide faster lookup. 1073 1074Removed ZCOPY emulation: The bdev module can be checked to see if it supports ZCOPY 1075and if not supported then use existing READ/WRITE commands. 1076 1077Added iov to spdk_bdev_zcopy_start. 1078 1079### dpdk 1080 1081Updated DPDK submodule to DPDK 21.05. 1082 1083### idxd 1084 1085Remove the `probe_cb` parameter in `spdk_idxd_probe` function. And remove the definition 1086of `spdk_idxd_probe_cb` function pointer. It should be implemented in idxd_user.c. 1087 1088Added API `spdk_idxd_submit_copy_crc32c` to perform a CRC32C while copying data. 1089 1090Added API `spdk_idxd_batch_prep_copy_crc32c` to prepare a batch operation to perform 1091a CRC32C while copying data. 1092 1093We have userspace idxd driver under accel_engine library (module/accel/idxd/accel_engine). 1094And kernel idxd driver usage is also added under the accel_engine library. This approach 1095implemented a wrapper library to use IDXD device by leveraging the kernel DSA driver in 1096SPDK idxd library (lib/idxd). Then users can leverage the RPC later to configure how to 1097use the DSA device by user space driver or kernel driver. 1098 1099### init 1100 1101Added new `init` library that initializes the SPDK subsystems, which previously was 1102internal to application framework. That functionality has been made public and for 1103use by applications which don't leverage SPDK's application framework. 1104 1105### iscsi 1106 1107New parameters, `pdu_pool_size`, `immediate_data_pool_size`, and `data_out_pool_size`, 1108were added to the RPC `iscsi_set_options` to run iSCSI target with varying amount of 1109available memory. 1110 1111### json 1112 1113Added API `spdk_json_write_named_uint128` and `spdk_json_write_uint128` to perform 1114the uint128 related data. 1115 1116### net 1117 1118Removed deprecated `net` library. 1119 1120Added support for ZCOPY. ZCOPY is used in preference to READ and WRITE if the 1121bdev module supports ZCOPY and the bdev module has ZCOPY enabled. 1122 1123### nvme 1124 1125`spdk_nvme_map_prps` and `spdk_nvme_map_cmd` were moved to nvmf/vfio-user as internal APIs 1126as vfio-user is the only user for the above two APIs. 1127 1128Added a new function `spdk_nvme_ns_cmd_copy` to submit a Simple Copy Command to a Namespace. 1129 1130Update the `spdk_nvme_generic_command_status_code` structure with new status code 1131according to the definition in NVMe 1.4 spec. 1132 1133`spdk_nvme_ctrlr_get_default_ctrlr_opts` now sets `use_cmb_sqs` to false. This means 1134that if a controller has a CMB and supports SQs in the CMB, SPDK will not use 1135the CMB for SQs by default - the user must set `use_cmb_sqs` to true in 1136the `spdk_nvme_ctrlr_opts` structure prior to controller attach. 1137 1138Add a new function `spdk_nvme_detach_poll` to simplify a common use case to continue 1139polling until all detachments complete. 1140 1141Added new argument `timeout_admin_us` to `spdk_nvme_ctrlr_register_timeout_callback` so callers 1142can specify a different timeout for admin commands vs. io commands. 1143 1144An existing function `spdk_nvme_detach_async` was updated to add one or more detachments 1145to an active context while it is being polled. 1146 1147Rename a variable in the member `cmic` of the struct `spdk_nvme_ctrlr_data` from 1148`multi_host` to `multi_ctrlr`. The variable means that the NVM subsystem may have two or 1149more controllers if set to 1. However `multi_host` had indicated a particular use case 1150such that the NVM subsystem is used by multiple hosts. 1151 1152A new option `disable_read_ana_log_page` was added to struct `spdk_nvme_ctrlr_opts` to disable 1153reading ANA log page. The upper layer is expected to read ANA log page instead if `true`. 1154The default value is `false`. 1155 1156New APIs, `spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async`, have been added to 1157reset a controller asynchronously. 1158 1159New RPC `bdev_nvme_reset_controller` was added, to reset an NVMe controller. 1160 1161Added `spdk_nvme_ns_get_nguid` function to get NGUID for the given namespace. 1162 1163Added `spdk_nvme_ctrlr_is_fabrics` function to indicate whether a ctrlr handle 1164is associated with a fabrics controller. 1165 1166### nvmf 1167 1168Added `min_cntlid` and `max_cntlid` to `nvmf_create_subsystem` to limit the controller ID range. 1169Added `spdk_nvmf_subsystem_get_min_cntlid` and `spdk_nvmf_subsystem_get_max_cntlid` to request those values. 1170 1171`spdk_nvmf_request_get_buffers_multi` API is removed. 1172 1173Added the `nvmf_set_crdt` RPC for setting command retry delay times. 1174 1175Expanded `spdk_nvmf_poll_group_stat` with current qpair count statistics. 1176 1177Removed following deprecated APIs: 1178 1179- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 1180- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 1181- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 1182- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 1183- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 1184- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 1185- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 1186 1187### rpc 1188 1189New RPC `bdev_rbd_register_cluster` and `bdev_rbd_unregister_cluster` was added, it allows to create 1190and delete the rados object cluster, then users can choose the cluster to create related rbd 1191device. 1192 1193Revised `bdev_rbd_create` parameter, it allows to use an optional parameter `--cluster-name` 1194to create a rbd bdev with an already registered Rados Cluster Object. 1195 1196New RPC `bdev_rbd_get_clusters_info` was added, it allows to get the info of the registered 1197Rados Cluster names. 1198 1199New optional parameter, `timeout_admin_us`, added to the bdev_nvme_set_options RPC. 1200 1201Revised a parameter `--stripe-size_kb` to `--stripe-size-kb` of `bdev_raid_create` method 1202provided in `scripts/rpc.py` for consistency. 1203 1204An new optional parameter `config_kernel_mode` was added to the RPC `idxd_scan_accel_engine`, 1205and this is used to enable using the kernel mode IDXD driver. 1206 1207Removed deprecated parameter `enable-zerocopy-send` of RPC `sock_impl_set_options`, 1208use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 1209Removed deprecated parameter `disable-zerocopy-send` of RPC `sock_impl_set_options`, 1210use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 1211 1212### rpm 1213 1214Removed deprecated `pkg/spdk.spec`, please use `rpmbuild/spdk.spec` instead. 1215See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 1216 1217### thread 1218 1219Red-black tree has been used for timed pollers to provide faster insertion and deletion 1220and for io_devices to provide faster lookup. 1221 1222Removed `spdk_io_channel` structure from public header and moved it to thread_internal.h. 1223 1224### trace 1225 1226Added `spdk_trace_register_description_ext` function to register variable number of tracepoint arguments. 1227 1228Added ability to chain multiple trace entries together to extend the size of the argument buffer. 1229 1230### util 1231 1232Red-black tree macros has been added by using the macros provided by the FreeBSD operating system 1233under the same BSD license. 1234 1235Add an new macro `SPDK_SIZEOF_MEMBER` to get the size of a member of a struct. 1236 1237`spdk_crc32c_iov_update` function was added to support calculating the crc32c of the iovs. 1238 1239Added zipf random number generator with power law probability distribution. 1240When applied to performance testing of block devices, it will select blocks over 1241the full range of LBAs, but will more frequently select lower-numbered LBAs. 1242 1243## v21.04 1244 1245### accel 1246 1247Two new accelerated crc32 functions `spdk_accel_submit_crc32cv` and 1248`spdk_accel_batch_prep_crc32cv` are added in order to provide the 1249chained accelerated CRC32 computation support. 1250 1251### bdev 1252 1253For `bdev_ocssd_create` RPC, the optional parameter `range` was removed. 1254Only one OCSSD bdev can be created for one OCSSD namespace. 1255 1256Removed the `spdk_bdev_open` from bdev library API. 1257Removed the `spdk_vbdev_register` and `spdk_bdev_part_base_construct` from bdev module API. 1258Removed the `config_text` function for bdev modules to report legacy config. 1259 1260Added `spdk_bdev_get_max_active_zones` API to display maximum number active zones of a given bdev. 1261 1262Added `spdk_bdev_get_max_zone_append_size` API to display maximum zone append data transfer size. 1263 1264### bdev_nvme 1265 1266Added support for zoned namespaces. 1267 1268### blobstore 1269 1270Removed the `spdk_bdev_create_bs_dev_from_desc` and `spdk_bdev_create_bs_dev` API. 1271 1272### env 1273 1274Added `spdk_pci_device_allow` API to allow applications to add PCI addresses to 1275the allowed list after the application has started. 1276 1277Removed the `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts`. 1278 1279Added hotplug support based on uevent in `pci_event.c`. Added usage of this functionality in 1280nvme, virtio-scsi and virtio_blk libraries. Please see the new API `spdk_pci_event_listen`, 1281`spdk_pci_get_event`, `spdk_pci_register_error_handler` and `spdk_pci_unregister_error_handler`. 1282 1283### event 1284 1285Removed the `config_file`, `max_delay_us`, `pci_whitelist` 1286and `pci_blacklist` members of struct `spdk_app_opts`. 1287 1288### idxd 1289 1290A new API `spdk_idxd_device_needs_rebalance` was added so that users of the library 1291can know whether they need to rebalance the flow control for the channel 1292that was just added/removed. This is based on how the low level library 1293shares devices amongst channels. 1294 1295The API `spdk_idxd_reconfigure_chan` had the `num_channels` removed as this 1296is now tracked in the library. The app makes use the new API above to 1297determine whether to rebalance or not. This applies to `spdk_idxd_configure_chan` 1298as well. 1299 1300The API `spdk_idxd_put_channel` now returns the rebalance state for the 1301underlying device. 1302 1303### iscsi 1304 1305A security vulnerability has been identified and fixed in the SPDK iSCSI target. 1306A TEXT PDU with no data, but CONTINUE flag set, would result in a NULL pointer dereference 1307and crash the SPDK iSCSI target process. All users of the SPDK iSCSI target 1308are recommended to update. All SPDK versions <= v21.01 are affected. 1309 1310### net 1311 1312The net library is deprecated and will be removed in the 21.07 release. 1313 1314### nvme 1315 1316Added a new function `spdk_nvme_ctrlr_get_regs_pmrcap` to get the PMR capabilities. 1317 1318Added an accelerated table pointer in `spdk_nvme_poll_group` 1319which can be used to provide the accelerated functions by users with 1320hardware engine, such as crc32c accelerated function. 1321 1322Added new functions `spdk_nvme_ctrlr_get_pmrsz`, `spdk_nvme_ctrlr_enable_pmr`, 1323`spdk_nvme_ctrlr_disable_pmr`, `spdk_nvme_ctrlr_map_pmr` and `spdk_nvme_ctrlr_unmap_pmr`. 1324 1325Added NVMe transport operations to enable, disable, map and unmap the PMR. 1326 1327Added `spdk_nvme_qpair_get_optimal_poll_group` function and `qpair_get_optimal_poll_group` 1328function pointer in spdk_nvmf_transport_ops structure in order to add the qpair to the most 1329suitable polling group. 1330 1331Added OPTPERF and namespace optimal performance fields to nvme_spec.h. 1332 1333Added `spdk_nvme_set_hotplug_filter` API to allow applications to choose which 1334hot-inserted SSDs should be probed. This is useful for use cases where multiple 1335independent SPDK processes are running on one node. The filter function can 1336then be implemented in these processes to decide which SSDs to probe based on 1337the new SSD's PCI address. 1338 1339New functions `spdk_nvme_poll_group_get_stats` and `spdk_nvme_poll_group_free_stats` 1340were added. These functions allow to get transport statistics per NVME poll group. 1341 1342Added `spdk_nvme_map_cmd` API to map the NVMe command with SGL cases. 1343 1344Added support for vector variant of ZNS zone append commands with new API 1345`spdk_nvme_zns_zone_appendv` and `spdk_nvme_zns_zone_appendv_with_md`. 1346 1347Added `spdk_nvme_zns_ns_get_max_open_zones` and `spdk_nvme_zns_ns_get_max_active_zones` API, 1348to display maximum number of open and active zones of the given namespace. 1349 1350Added `spdk_nvme_zns_ns_get_zone_size_sectors` API to provide size of zone in number of 1351sectors. 1352 1353Added `spdk_nvme_qpair_get_id` API to display the ID of the specified qpair. 1354 1355### nvmf 1356 1357Removed the `spdk_nvmf_tgt_listen` and `spdk_nvmf_subsystem_add_ns` API. 1358 1359Added new APIs: 1360 1361- `spdk_nvmf_poll_group_dump_stat` (function in `nvmf.h`). 1362- `poll_group_dump_stat` (transport op in `nvmf_transport.h`). 1363 1364The following APIs have been deprecated and will be removed in SPDK 21.07: 1365 1366- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 1367- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 1368- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 1369- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 1370- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 1371- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 1372- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 1373 1374See header files for details. 1375 1376The `trtype` field in JSON returned by `nvmf_get_stats` RPC contains now the name of the transport, 1377which is the same as the type for defined transports and more informative for a custom transport. 1378 1379Added `hdgst` and `ddgst` parameters to `bdev_nvme_attach_controller` RPC in order change 1380state of TCP header and data digest. 1381 1382Added `num_cqe` parameter to `nvmf_create_transport` RPC to set number of completion queues (CQ) 1383for RDMA transport. Useful when CQ resize operation is not supported, for example iWARP. 1384 1385### ocf 1386 1387Updated OCF submodule to v20.12.2 1388 1389Added `bdev_ocf_set_cache_mode` RPC to dynamically switch cache mode of OCF bdev. 1390 1391OCF deprecation notice has removed as Huawei is picking up support for the OCF project. 1392 1393### opal 1394 1395Removed the `spdk_opal_supported` API. 1396 1397### raid 1398 1399For `bdev_raid_create` RPC, the deprecated parameter `strip_size` was removed. 1400 1401### rpc 1402 1403New RPC `bdev_nvme_get_transport_statistics` was added, it allows to get transport statistics 1404of nvme poll groups. 1405 1406Parameter `enable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1407use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 1408Parameter `disable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1409use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 1410 1411Added cmd_parser.py used to parse the args from argparse. There are 1412two benefit to use command parser: 1413 1414- Simplify the definition of rpc method. It will reduce the rpc method code. 1415- Make the rpc call more versatile. User can add private args into rpc method. 1416 1417### rpm 1418 1419Added support for new RPM spec, rpmbuild/spdk.spec, which can be used for packaging the 1420SPDK. The pkg/spdk.spec is considered to be deprecated and scheduled for removal in SPDK 21.07. 1421See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 1422 1423### sock 1424 1425The type of `enable_placement_id` in struct `spdk_sock_impl_opts` is changed from 1426bool to int. We can use RPC to configure different value of `enable_placement_id`. 1427Then we can leverage SO_INCOMING_CPU to get placement_id, which aims to utilize 1428CPU cache locality, enabled by setting enable_placement_id=2. 1429 1430A new socket placement mode called PLACEMENT_MARK has been added. Some NICs allow 1431sockets to be marked using the SO_MARK socket option as a hint for which hardware 1432queue they should be associated with. This mode leverages that by setting the same 1433value for all sockets within a poll group. 1434 1435New parameters `enable_zerocopy_send_server` and `enable_zerocopy_send_client` were added 1436to struct spdk_sock_impl_opts, these parameters enable or disable zero copy send for server 1437and client sockets which are created using `spdk_sock_listen` and `spdk_sock_listen_ext` (server); 1438`spdk_sock_connect` and `spdk_sock_connect_ext` (client) functions. Existing parameter 1439`enable_zerocopy_send` enables or disables zero copy send for both server and client sockets. 1440 1441### thread 1442 1443A new API `spdk_io_channel_get_io_device` was added to get the io_device for the specified 1444I/O channel. 1445 1446Added `spdk_thread_set_interrupt_mode` function in order to set present spdk_thread into 1447interrupt mode or back to poll mode. It is valid only when thread interrupt facility is 1448enabled by `spdk_interrupt_mode_enable`. 1449 1450Added `spdk_poller_register_interrupt` function to mark that the poller is capable of 1451entering interrupt mode. Callback function will be called when the poller must transition 1452into or out of interrupt mode. 1453 1454### virtio 1455 1456Added the `bdev_virtio_blk_set_hotplug` RPC for the virtio blk pci device. 1457 1458## v21.01 1459 1460### bdev 1461 1462An `opts_size` element was added in the `spdk_bdev_opts` structure to solve the 1463ABI compatibility issue between different SPDK version. And also add `opts_size` 1464parameter in spdk_bdev_get_opts function. Two fields `small_buf_pool_size` and 1465`large_buf_pool_size` were added into spdk_bdev_opts, which were used to determine 1466the small and large buffer pool size of the whole bdev module. 1467 1468A new API `spdk_bdev_wait_for_examine` was added to allow for checking state of 1469examine process. Along with corresponding `bdev_wait_for_examine` RPC, which 1470is now always called during `spdk_bdev_subsystem_config_json` making sure 1471every bdev is ready to be used. 1472 1473A new API `spdk_bdev_io_get_aio_status` was added for getting the status of 1474bdev_io as Linux AIO errno. Also `spdk_bdev_io_complete_aio_status` function 1475and `SPDK_BDEV_IO_STATUS_AIO_ERROR` were added for bdev module to complete 1476a bdev_io with Linux AIO errno. 1477 1478A new API `spdk_bdev_get_module_name` was added for getting block device module name. 1479Also `spdk_bdev_get_module_ctx` was added to obtain a bdev module context for 1480the block device opened by the specified descriptor. 1481 1482Added `max_segment_size` and `max_num_segments` to `spdk_bdev` structure for 1483bdev modules to specify splitting requirements. 1484 1485### blob 1486 1487An `opts_size` element was added in the `spdk_bs_opts` structure to solve the 1488ABI compatibility issue between different SPDK version. And also add `opts_size` 1489parameter in `spdk_bs_opts_init` function. 1490 1491An `opts_size` element was added in the `spdk_blob_opts` structure to solve the 1492ABI compatibility issue between different SPDK version. And also add `opts_size` 1493parameter in `spdk_blob_opts_init` function. 1494 1495An `opts_size` element was added in the `spdk_blob_open_opts` structure to solve the 1496ABI compatibility issue between different SPDK version. And also add `opts_size` 1497parameter in `spdk_blob_open_opts_init` function. 1498 1499### build 1500 1501SPDK now generates pkg-config files to simplify the process of determining which 1502libraries must be linked into an SPDK application. 1503 1504### dpdk 1505 1506Updated DPDK submodule to DPDK 20.11. 1507 1508Removed `--with-igb-uio-driver` configure option. Since DPDK 20.11 igb_uio driver 1509was moved to separate dpdk-kmods repository. If required, users need to refer to 1510dpdk-kmods repository to build the driver. 1511 1512### env 1513 1514The `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts` 1515have been deprecated. The new members are named `pci_allowed`, `pci_blocked` and 1516`main_core` respectively. 1517 1518An `opts_size` element was added in the `spdk_app_opts` structure 1519to solve the ABI compatibility issue between different SPDK version. An `opts_size` 1520parameter is added into `spdk_app_opts_init` function. 1521 1522### event 1523 1524The `pci_whitelist` and `pci_blacklist` members of struct `spdk_app_opts` have been 1525deprecated. The new members are named `pci_allowed` and `pci_blocked` respectively. 1526 1527The `--pci-blacklist` command line option has been deprecated, replaced with 1528`--pci-blocked`. 1529 1530The `--pci-whitelist/-W` command line options have been deprecated, replaced with 1531`--pci-allowed/-A`. 1532 1533Added new experimental `dynamic` scheduler that rebalances idle threads, adjusts CPU frequency 1534using dpdk_governor and turns idle reactor cores to interrupt mode. Please see 1535[scheduler documentation](https://www.spdk.io/doc/scheduler.html) for details. 1536 1537## ioat 1538 1539The PCI BDF whitelist option has been removed from the `ioat_scan_accel_engine` RPC. 1540IOAT PCI functions can still be allowed or denied using SPDK application command 1541line options. 1542 1543### isa-l 1544 1545Updated ISA-L submodule to v2.30.0. 1546 1547### json 1548 1549A new API `spdk_jsonrpc_send_bool_response` was added to allow sending response for 1550writing json bool results into one function. 1551Update API `bdev_nvme_set_options` and add a `keep_alive_timeout_ms` parameter. Now you 1552can specify the `keep_alive_timeout` before creating NVMe bdev. 1553 1554### nbd 1555 1556Change the return type of function `spdk_nbd_stop` from void to int. And update the 1557`spdk_nbd_fini` with two parameters to make its behavior from sync to async. 1558 1559### nvme 1560 1561Directives support was added to the NVMe driver. 1562 1563Two async APIs `spdk_nvme_ctrlr_cmd_directive_receive` and `spdk_nvme_ctrlr_cmd_directive_send` 1564are added for Directive Send and Directive Receive command, respectively. 1565 1566Added a new function `spdk_nvme_ctrlr_reset_subsystem` to perform a NVMe 1567subsystem reset. Note: The NVMe-oF target does not support the subsystem reset yet. 1568 1569Add a new function `spdk_nvme_bytes_to_numd` to transfer bytes to number of 1570dwords. 1571 1572Added a new custom transport `SPDK_NVME_TRANSPORT_VFIOUSER` to enable NVMe 1573driver running with NVMe over vfio-user target. 1574 1575Added the vfio-user custom transport implementation in NVMe driver which can connect 1576to NVMe over vfio-user target via vfio-user transport. 1577 1578Added a new function `spdk_nvme_ctrlr_set_remove_cb` to remove or override 1579`remove_cb` and `remove_ctx` specified when the controller was probed. 1580 1581Added support for ZNS zone append command with new API `spdk_nvme_zns_zone_append` and 1582`spdk_nvme_zns_zone_append_with_md`. 1583 1584### nvmf 1585 1586`nvmf_fc_lld_fini` now takes callback and hence updating FC Broadcom LLD driver 1587to the latest is required. 1588 1589FC transport supported primary tagging and VMID. 1590 1591Broadcom FC LLD driver and SPDK NVMe-oF FC transport consolidated one LLD API, 1592`nvmf_fc_init_poller_queues` into another LLD API `nvmf_fc_init_q`. 1593Hence updating Broadcom FC LLD driver to the latest is required. 1594 1595The functions `destroy` and `qpair_fini` in the transport interface now accept a 1596`cb_fn` and `cb_arg` to call upon completion, and their execution can be asynchronous. 1597 1598The SPDK NVMe-oF target now supports async event notification for discovery log changes. 1599This allows the initiator to create persistent connection to discovery controller and 1600be notified of any discovery log changes. 1601 1602An `opts_size` element was added in the `spdk_nvmf_transport_opts` structure 1603to solve the ABI compatibility issue between different SPDK version. And also add 1604`opts_size` parameter in `spdk_nvmf_transport_opts_init` function. 1605 1606Added a new custom vfio-user transport implementation in NVMe-oF which can provide 1607emulated NVMe devices to QEMU and SPDK NVMe driver. 1608 1609Added new API `spdk_nvmf_tgt_listen_ext` that uses `spdk_nvmf_listen_opts` to allow 1610passing generic options to specific transport. This functionality replaces one provided 1611with `nvmf_subsystem_set_options`. Also removed `nvmf_subsystem_set_options` RPC 1612that allowed transport specific options within subsystem. 1613 1614The `trsvcid` in `nvmf_subsystem_add_listener` RPC is now optional. 1615 1616Pausing a subsystem now only pauses all admin queues. This allows for the 1617management of hosts and listeners, plus the addition of namespaces without a 1618full subsystem pause. Additionally, the target now allows for pausing 1619individual namespaces within a subsystem. To remove a namespace from a 1620subsystem, only the specific namespace must be paused. I/O will continue to 1621other namespaces while these operations execute. 1622 1623### rpc 1624 1625An new optional parameter `wait` was added to the RPC `iscsi_create_portal_group`, 1626and an new RPC `iscsi_start_portal_group` was added. They will be used not to start 1627listening on portals for a portal group until all associated target nodes are created 1628at startup, otherwise some iSCSI initiators may fail to re-login when SPDK iSCSI 1629target application restarts. 1630 1631Two optional parameter `--small-buf-pool-size` and `--large-buf-pool-size` were added 1632into `bdev_set_options` function. 1633 1634### vhost 1635 1636Added optional `packed_ring_recovery` parameter to `vhost_create_blk_controller` RPC 1637enabling packed ring live recovery. 1638 1639## v20.10 1640 1641### accel 1642 1643New APIs `spdk_accel_batch_cancel` and `spdk_idxd_batch_cancel` were added to allow 1644cancelling a batch sequence. 1645 1646### bdev 1647 1648A new `spdk_bdev_part_base_construct_ext` function has been added and the 1649`spdk_bdev_part_base_construct` has been deprecated. The 1650`spdk_bdev_part_base_construct_ext` function takes bdev name as an argument instead 1651of bdev structure to avoid a race condition that can happen when the bdev is being 1652removed between a call to get its structure based on a name and actually opening it. 1653 1654Removed `spdk_bdev_config_text` function for bdev modules to report legacy config. 1655 1656### blobstore 1657 1658A new `spdk_bdev_create_bs_dev_ext` function has been added and `spdk_bdev_create_bs_dev_from_desc` 1659function has been deprecated. 1660 1661### dpdk 1662 1663Updated DPDK submodule to DPDK 20.08. 1664 1665### env 1666 1667A new API `spdk_mem_get_fd_and_offset` was added to get file descriptor and offset for 1668a given virtual address. 1669 1670### event 1671 1672Removed `spdk_subsystem_config` callback for submodules as part of legacy config removal. 1673 1674Removed `spdk_app_get_running_config` function that printed configuration in legacy format, 1675and removed `usr1_handler` from `struct spdk_app_opts` callback that was used to call it. 1676 1677Added SPDK thread scheduler framework which is used to rebalance load among CPU cores. 1678The scheduler implementation is pluggable and two schedulers are added first, `static` scheduler 1679and `gscheduler` scheduler. 1680 1681`static` scheduler is used by default and does not reschedule threads and keeps the initial 1682assignments. 1683 1684CPU frequency governor framework and implementation are also added. 1685`dpdk_governor` is the first implementation and is based on rte_power library of DPDK. 1686`gscheduler` scheduler utilizes the CPU frequency governor. 1687 1688Scheduler and governor frameworks are experimental features. 1689 1690### fio 1691 1692Bdev fio_plugin no longer supports legacy configuration files. Options `spdk_conf` and 1693`spdk_json_conf` have to point to valid JSON configuration. 1694 1695### gpt 1696 1697Removed option to disable GPT probing via legacy configuration. Please use generic 1698bdev functionality to replicate the behavior. See `bdev_set_options --disable-auto-examine` 1699and `bdev_examine` RPC. 1700 1701### intel-ipsec-mb 1702 1703Updated intel-ipsec-mb submodule to v0.54 1704 1705### isa-l 1706 1707Updated ISA-L submodule to v2.29.0. 1708 1709### iscsi 1710 1711The iSCSI target supports iSCSI login redirection feature to achieve fault tolerance 1712and load balancing particularly in scale out iSCSI target system, which runs multiple 1713SPDK iSCSI targets. 1714 1715### json 1716 1717A new API `spdk_json_decode_object_relaxed` was added to allow parsing json values 1718even if decoder for a given key is not found. 1719 1720A new API `spdk_json_free_object` was added to free memory allocated by `spdk_json_decode_object`. 1721 1722### log 1723 1724A log flag structure, `spdk_log_register_flag`, `spdk_log_get_first_flag`, 1725`spdk_log_get_next_flag` and macro `SPDK_LOG_REGISTER_COMPONENT`, have been added which 1726enable registering and interacting with log flags. Added `SPDK_INFOLOG`, `SPDK_DEBUGLOG` 1727and `SPDK_LOGDUMP` macros to display logs for specified log flag. 1728 1729The log flags in the event framework `-L` and several SPDK applications has been enabled on 1730release builds of SPDK. On debug builds this option will additionally set 1731log print level to `SPDK_LOG_DEBUG`. 1732 1733A new API `spdk_log_enable_timestamps` and new RPC `log_enable_timestamps` was added to 1734toggle log timestamps. 1735 1736### miscellaneous 1737 1738The contents of the log_rpc library have been moved to the event library. The log_rpc 1739library now no longer exists. 1740 1741The contents of the app_rpc library have been moved to the event library. The app_rpc 1742library now no longer exists. 1743 1744The contents of the bdev_rpc library have been moved to the bdev library. The app_rpc 1745library now no longer exists. 1746 1747The bdevperf application now disables the zcopy API by default. Prior to this change, 1748bdevperf enabled using the zcopy API by default which caused a performance impact of 1749up to 25% on bdevs that don't natively support zcopy because the API emulates zero-copy 1750by allocating a buffer. The bdevperf `-x` param was renamed to `-Z` and the default 1751value changed to false. For bdevs that support zcopy, use the -Z flag to enable 1752using zcopy API. 1753 1754### net 1755 1756Net framework intended for use by user-space TCP stacks was removed. 1757 1758### nvme 1759 1760Zoned Namespace Command Set support has been added to NVMe driver, see `nvme_zns.h` for details. 1761 1762New APIs, `spdk_nvme_detach_async` and `spdk_nvme_detach_poll_async`, have been added to 1763detach multiple controllers in parallel to mitigate lengthy shutdown notification of 1764a few NVMe SSDs. 1765 1766New APIs, `spdk_nvme_ns_get_ana_group_id` and `spdk_nvme_ns_get_ana_state`, have been 1767added to get ANA group ID and ANA state of the given namespace, respectively. 1768 1769If no specific command set is requested (by setting the command_set member in the 1770`spdk_nvme_ctrlr_opts` structure), SPDK will automatically select the most appropriate 1771command set based on what the controller supports. 1772 1773### nvmf 1774 1775A new API `spdk_nvmf_subsystem_add_ns_ext` has been added and the API `spdk_nvmf_subsystem_add_ns` 1776has been deprecated. 1777 1778The NVMe-oF target now supports Asymmetric Namespace Access (ANA) Reporting to provide 1779multipath to NVMe-oF initiator. 1780 1781Add `no_wr_batching` parameter to `spdk_nvmf_transport_opts` struct to disable 1782Work Requests batching in RDMA transport. 1783 1784NVMf Target transports can now parse any additional JSON params in the nvmf_create_transport RPC 1785through the JSON context provided via spdk_nvmf_target_opts->transport_specific. 1786 1787A new RPC `nvmf_subsystem_set_options` was added to allow transport specific options within subsystem. 1788 1789A new API `spdk_nvmf_transport_stop_listen_async` was added to stop accepting new connections 1790at the provided address. `nvmf_subsystem_remove_listener` RPC now uses this function. 1791 1792A new API `spdk_nvmf_subsystem_disconnect_host` was added to disconnect all connections 1793originating from the provided hostnqn. `nvmf_subsystem_remove_host` RPC now uses this function. 1794 1795### ocf 1796 1797Updated OCF submodule to v20.03.1 1798 1799### rpc 1800 1801New optional parameters, `enable_placement_id` and `enable_quickack` were added to the 1802`sock_impl_set_options` RPC. 1803 1804A new RPC `bdev_examine_bdev` was added to allow users to examine a bdev explicitly. 1805It can be used only if bdev_auto_examine is set to false by the RPC `bdev_set_options`. 1806 1807New optional parameters `no_wr_batching` and `control_msg_num` were added to the RPC 1808'nvmf_create_transport'. 1809 1810New RPCs, `iscsi_target_node_set_redirect` and `iscsi_target_node_request_logout`, have 1811been added, and a new optional parameter `private` has been added to the RPC 1812`iscsi_create_portal_group` to support iSCSI login redirection feature. 1813 1814A new optional parameter `ana_reporting` was added to the RPC `nvmf_create_subsystem`, and 1815a new RPC `nvmf_subsystem_listener_set_ana_state` was added for ANA reporting. 1816 1817New RPCs, `nvmf_subsystem_get_listeners` and `nvmf_subsystem_get_qpairs`, were added to 1818retrieve configuration of the NVMe-oF subsystem. 1819 1820### scsi 1821 1822Two new APIs have been added `spdk_scsi_dev_construct_ext` and 1823`spdk_scsi_dev_add_lun_ext` that allow the upper layer(e.g. vhost-scsi) to 1824receive the notification when the scsi bdev has been resized. 1825 1826The `spdk_scsi_dev_construct` and `spdk_scsi_dev_add_lun` eventually may be 1827deprecated and removed. 1828 1829### sock 1830 1831The `enable_placement_id` field was added in the `struct spdk_sock_impl_opts` to 1832make the placement_id feature configurable by users. The default setting is 1833not enabled. 1834 1835The `enable_quick_ack` field was added in the `struct spdk_sock_impl_opts` to enable 1836or disable quick ack for the POSIX sock module. The default setting is not enabled. 1837 1838### thread 1839 1840`fd_group` is applied to support interrupt mode. 1841New APIs were added to support an experimental interrupt mode. This allows modules or 1842libraries to selectively register file descriptors that the spdk_thread can wait on, 1843as an alternative to polling. In v20.10, this functionality is enabled in a very small 1844subset of SPDK libraries and modules. 1845 1846### util 1847 1848A new utility named `fd_group` was add. It is now 1849implemented by epoll on Linux platform. It can be used by 1850spdk_thread and reactor to implement interrupt mode. 1851 1852Added `spdk_bit_pool` utility, see `bit_pool.h` for details. 1853 1854### vhost 1855 1856SPDK has switched to DPDK's rte_vhost library since 19.07 release, removed the internal 1857rte_vhost library which is used for DPDK older than 19.05, removed the experimental vhost 1858nvme target which depends on the internal rte_vhost library. 1859 1860### vpp 1861 1862Removed socket abstraction for VPP. 1863 1864## v20.07: SPDK CSI driver, new accel_fw commands, I/O abort support 1865 1866spdk_json_free_object() was added to free memory allocated by spdk_json_decode_object(). 1867 1868### accel 1869 1870A new API was added `spdk_accel_get_capabilities` that allows applications to 1871query the capabilities of the currently enabled accel engine back-end. 1872 1873A new capability, CRC-32C, was added via `spdk_accel_submit_crc32c`. 1874 1875The software accel engine implementation has added support for CRC-32C. 1876 1877A new capability, compare, was added via `spdk_accel_submit_compare`. 1878 1879The software accel engine implementation has added support for compare. 1880 1881Several APIs were added to `accel_engine.h` to support batched submission 1882of operations. 1883 1884Several APIs were added to `accel_engine.h` to support dualcast operations. 1885 1886### accel_fw 1887 1888The accel_fw was updated to no longer require the app to allocate an 1889accel_task on its behalf. All public APIs now take a callback arg as 1890the parameter that used to be the accel_task. 1891 1892The accel_fw API `spdk_accel_task_size` was removed as it is no longer 1893required. 1894 1895The accel_fw was updated to support compare, dualcast, crc32c. 1896 1897The accel_fw introduced batching support for all commands in all plug-ins. 1898See docs for detailed information. 1899 1900### bdev 1901 1902A new API `spdk_bdev_abort` has been added to submit abort requests to abort all I/Os 1903whose callback context match to the bdev on the given channel. 1904 1905### build 1906 1907The fio plugins now compile to `build/fio` and are named `spdk_bdev` and `spdk_nvme`. 1908Existing fio configuration files will need to be updated. 1909 1910### dpdk 1911 1912Updated DPDK submodule to DPDK 20.05. 1913 1914### env 1915 1916Several new APIs have been added to provide greater flexibility in registering and 1917accessing polled mode PCI drivers. See `env.h` for more details. 1918 1919### idxd 1920 1921The idxd library and plug-in module for the accel_fw were updated to support 1922all accel_fw commands as well as batching. Batching is supported both 1923through the library and the plug-in module. 1924 1925IDXD engine support for CRC-32C has been added. 1926 1927### ioat 1928 1929A new API `spdk_ioat_get_max_descriptors` was added. 1930 1931### nvme 1932 1933An `opts_size`element was added in the `spdk_nvme_ctrlr_opts` structure 1934to solve the ABI compatibility issue between different SPDK version. 1935 1936A new API `spdk_nvme_ctrlr_cmd_abort_ext` has been added to abort previously submitted 1937commands whose callback argument match. 1938 1939Convenience functions, `spdk_nvme_print_command` and `spdk_nvme-print_completion` were added 1940to the public API. 1941 1942A new function, `spdk_nvmf_cuse_update_namespaces`, updates the cuse representation of an NVMe 1943controller. 1944 1945A new function `qpair_iterate_requests` has been added to the nvme transport interface. ALl 1946implementations of the transport interface will have to implement that function. 1947 1948### nvmf 1949 1950The NVMe-oF target no longer supports connecting scheduling configuration and instead 1951always uses what was previously called "transport" scheduling. 1952 1953`spdk_nvmf_tgt_accept` no longer exists. The accept process now occurs automatically after 1954the creation of an nvmf target and queue pairs are assigned to poll groups by the underlying 1955transport. Further, `spdk_nvmf_transport_ops` has changed such that the accept function 1956pointer no longer takes a function pointer as an argument. Instead, transports should call 1957`spdk_nvmf_tgt_new_qpair` whenever they previously would have called that callback. 1958 1959The NVMe-oF target now supports aborting any submitted NVM or Admin command. Previously, 1960the NVMe-oF target could abort only Asynchronous Event Request commands. 1961 1962### rdma 1963 1964A new `rdma` library has been added. It is an abstraction layer over different RDMA providers. 1965Two providers are available - verbs (used by default when RDMA is enabled or enabled explicitly 1966using --with-rdma=verbs) and mlx5 Direct Verbs aka DV (enabled by --with-rdma=mlx5_dv). 1967Using mlx5_dv requires libmlx5 installed on the system. 1968 1969### rpc 1970 1971Parameter `-p` or `--max-qpairs-per-ctrlr` of `nvmf_create_transport` RPC command accepted by the 1972rpc.py script is deprecated, new parameter `-m` or `--max-io-qpairs-per-ctrlr` was added. 1973 1974Added `sock_impl_get_options` and `sock_impl_set_options` RPC methods. 1975 1976Command line parameters `-r` and `--rpc-socket` will longer accept TCP ports. RPC server 1977must now be started on a Unix domain socket. Exposing RPC on the network, as well as providing 1978proper authentication (if needed) is now a responsibility of the user. 1979 1980The `bdev_set_options` RPC has a new option, `bdev_auto_examine` to control the auto examine function 1981of bdev modules. 1982 1983New RPCs `sock_impl_get_options` and `sock_impl_set_options` been added to expose new socket features. 1984See `sock` section for more details. 1985 1986### sock 1987 1988Added `spdk_sock_impl_get_opts` and `spdk_sock_impl_set_opts` functions to set/get socket layer configuration 1989options. Options can be set independently for each implementation. 1990 1991Added `recv_buf_size` and 'send_buf_size' socket layer options. They are used only in posix implementation. 1992 1993Added `uring` based socket implementation, the code is located in module/sock/uring. This feature is only 1994available in Linux which requires kernel version is greater than 5.4.3. Currently, our CI pool added the uring 1995based socket tests for iSCSI target and also the tests for SPDK NVMe-oF tcp transport. 1996 1997Added `enable_recv_pipe` socket layer option to allow disabling of double buffering on receive. 1998New option is used only in posix implementation. 1999 2000Added `enable_zerocopy_send` socket layer option to allow disabling of zero copy flow on send. 2001New option is used only in posix implementation. 2002 2003### util 2004 2005Some previously exposed CRC32 functions have been removed from the public API - 2006`spdk_crc32_update`, `spdk_crc32_table_init`, and the `spdk_crc32_table` struct. 2007 2008### vhost 2009 2010The function `spdk_vhost_blk_get_dev` has been removed. 2011 2012## v20.04: SPDK Top, IDXD, NVMe qpair groups 2013 2014IDXD engine support for compare has been added. 2015 2016### configuration 2017 2018Legacy INI style configuration for SPDK applications has been deprecated and will be 2019removed in future release. 2020Please switch to JSON-RPC configuration files and/or RPC driven run-time configuration. 2021 2022### copy 2023 2024The copy engine library, modules and public APIs have been renamed. Use of the word `copy` 2025has been replaced with the word `accel` short for accelerator in preparation for adding new 2026capabilities in the future. Additionally, APIs for what was previously called the `memcpy` 2027engine have been renamed to identify the engine as a software accelerator. 2028 2029### crypto 2030 2031Support for AES_XTS was added for the QAT polled mode driver (pmd). The create RPC 2032`bdev_crypto_create` has 2 new optional parameters: cipher and key2. Cipher can be either 2033AES_CBC (default) or AES_XTS. AES_XTS is only valid when using the QAT polled mode driver. 2034The key2 parameter is the second key required for AES_XTS. 2035 2036### event 2037 2038Reactors now accumulate CPU stats and they are retrieved by the RPC `framework_get_reactors`. 2039 2040### idxd 2041 2042IDXD support was added in the form of a low level library that can directly 2043utilize the IDXD offload engine as well as an accel plug-in module allowing for use 2044with the generic accel framework API. IDXD is the first in a family of offload 2045engines that share the same interface, specifically DSA is added here. More info 2046can be found here: https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator 2047 2048Much of the implementation models IOAT, however the low level interfaces are very 2049different. The RPC to enable IDXD requires a configuration number as well. The 2050code includes two pre-defined configurations of IDXD groups/work queues/engines. A future 2051version will provide an interface to allow for setting of individual configuration 2052parameters. 2053 2054IDXD is not yet available so this feature should be considered experimental. It will 2055be built up with additional documentation as an ongoing activity. 2056 2057### iscsi 2058 2059The iSCSI target now creates a lightweight thread per poll group instead of assuming a pool 2060of lightweight threads already exist at start up time. A poll group is a collection of 2061unrelated iSCSI connections. Each poll group is only accessed from the associated 2062lightweight thread. 2063 2064### ftl 2065 2066Several changes have been made to the `spdk_ftl_conf`, `spdk_ftl_dev_init_ops`, and 2067`spdk_ftl_attrs` structs. Please see `include/spdk/ftl.h` for more details. 2068 2069### miscellaneous 2070 2071The `--json-ignore-init-errors` command line parameter has been added to ignore 2072initialization errors on JSON config load. 2073 2074The public header file io_channel.h has been removed. Please use thread.h which has the 2075exact same API. 2076 2077### nvme 2078 2079Exported internal nvme_ctrlr_cmd_security_receive/send() APIs as public APIs with "the spdk_" 2080prefix. 2081 2082Added `priority` field in `spdk_nvme_transport_id`, this field is used to specify the priority 2083of the NVMe-oF connection, and currently it is used for NVMe-oF tcp connection. 2084 2085A new poll group API has been added to allow for pooling of nvme qpairs across a single 2086entity which can be polled for completions. This new API consists of the `spdk_nvme_poll_group` 2087family of functions. As a result of this new API, all NVMe transports are expected to implement 2088several poll group related functions. 2089 2090A new flag, `create_only`, has been added to the `spdk_nvme_io_qpair_opts` structure. This flag 2091allows a user to call `spdk_nvme_ctrlr_get_default_io_qpair` without also connecting the qpair 2092within the context of that call. 2093 2094As a result of the `create_only` flag, two new API functions, `spdk_nvme_ctrlr_connect_io_qpair` 2095and `spdk_nvme_ctrlr_disconnect_io_qpair`, have been added to facilitate connecting newly created 2096qpairs (for example, after they have been added to a poll group) and disconnecting qpairs without 2097destroying them (for example to disconnect a qpair before migrating it to a new poll group and 2098reconnecting it). 2099 2100The functions `spdk_nvme_ctrlr_alloc_cmb_io_buffer` and `spdk_nvme_ctrlr_free_cmb_io_buffer` 2101have been changed to `spdk_nvme_ctrlr_map_cmb` and `spdk_nvme_ctrlr_unmap_cmb` respectively. 2102 2103An additional function, `spdk_nvme_ctrlr_reserve_cmb`, has been added to facilitate reserving 2104the entire size of the controller memory buffer for data transfer. 2105 2106### nvme_cuse 2107 2108`spdk_nvme_cuse_get_ctrlr_name` now takes two additional parameters, `char *name` which 2109stores the pointer to the controller name, and `size_t *size` which stores the length of 2110the name. The return type has also been changed from char * to int. 2111 2112`spdk_nvme_cuse_get_ns_name` now takes two additional parameters, `char *name` which 2113stores the pointer to the namespace name, and `size_t *size` which stores the length of 2114the name. The return type has also been changed from char * to int. 2115 2116### nvme_opal 2117 2118Several public OPAL structure definitions have been changed since the last release. 2119 2120### nvmf 2121 2122`spdk_nvmf_poll_group_destroy()` is now asynchronous and accepts a completion callback. 2123 2124The NVMe-oF target now creates a lightweight thread per poll group instead of assuming a pool 2125of lightweight threads already exist at start up time. A poll group is a collection of 2126unrelated NVMe-oF connections. Each poll group is only accessed from the associated 2127lightweight thread. 2128 2129A new struct, `spdk_nvmf_subsystem_listener`, has been added to encapsulate the subsystem specific 2130nature of a listener object. 2131 2132`spdk_nvmf_tgt_listen` no longer accepts a callback function or argument. It also returns an 2133int to indicate the status of the listen call. 2134 2135The execution of `spdk_nvme_poll_group_destroy` is now asynchronous and the function accepts 2136a cb_fn and cb_arg to call upon completion. 2137 2138The execution of `spdk_nvmf_subsystem_add_listener` is now asynchronous and the function accepts 2139a cb_fn and cb_arg to call upon completion. 2140 2141The `nvmf_transport.h` header has been made public to allow custom NVMe-oF transports to integrate 2142with NVMe-oF libraries without using internal APIs. 2143 2144### ocf 2145 2146Updated the OCF submodule to OCF v20.03 2147 2148New version of OCF comes with API changes and bug fixes 2149 2150### rpc 2151 2152A new RPC `thread_set_cpumask` has been added to set the cpumask of the thread 2153to the specified value. 2154 2155A new RPC `thread_get_pollers` has been added to retrieve pollers from SPDK threads. 2156 2157A new RPC `thread_get_io_channels` has been added to retrieve I/O channels from SPDK threads. 2158 2159A new RPC `bdev_rbd_resize` has been added to resize Ceph RBD bdevs. 2160 2161### sock 2162 2163The `spdk_sock_set_priority` function has been removed since the feature to set the sock priority 2164will be contained in two new functions, i.e., `spdk_sock_listen_ext` and `spdk_sock_connect_ext`. 2165Users may now specify the priority of the socket in the opts that they want to use. 2166 2167### spdk_top 2168 2169A new application, `spdk_top`, has been added which allows users to monitor resource consumption 2170by a running SPDK application. More information on this application can be found in 2171`app/spdk_top/README`. 2172 2173### thread 2174 2175A new function `spdk_thread_lib_init_ext` has been added, and the function 2176`spdk_thread_lib_init` has been deprecated. The user of `spdk_thread_lib_init_ext` is 2177expected to implement both functions `spdk_thread_op_fn` and `spdk_thread_op_supported_fn`. 2178`spdk_thread_op_supported_fn` is called to check whether the SPDK thread operation 2179is supported. `spdk_thread_op_fn` is called to execute the SPDK thread operation. 2180Current SPDK operation types are `SPDK_THREAD_OP_NEW` and `SPDK_THREAD_OP_RESCHED`. 2181The operation `SPDK_THREAD_OP_NEW` is called each time a new thread is created. 2182The operation `SPDK_THREAD_OP_RESCHED` is called when SPDK thread needs to be rescheduled. 2183 2184A unique ID has been added for each created SPDK thread, it is retrieved by a new function 2185`spdk_thread_get_id`, and the SPDK thread which has the specific ID is got by 2186a new function `spdk_thread_get_by_id`. 2187 2188A new function `spdk_thread_cpumask` has been added to set the current thread's cpumask 2189to the specified value. The function requires the operation `SPDK_THREAD_OP_RESCHED` 2190is supported. 2191 2192A new function `spdk_poller_register_named` has been added to set arbitrary name to the 2193created poller. If NULL, the name is set to the pointer of the poller function. 2194 2195The function `spdk_thread_poll` now measures run time per call correctly on multiple SPDK 2196threads configuration, and a new function `spdk_thread_get_last_tsc` has been added to use together. 2197 2198Voluntary termination of SPDK thread has been supported by refining the functions `spdk_thread_exit` 2199and `spdk_thread_poll`. 2200 2201### util 2202 2203New functions `spdk_sn32_lt` and `spdk_sn32_gt` have been added. They compare two sequence 2204numbers based on serial number arithmetic. 2205 2206### vhost 2207 2208Poll groups per session have been replaced by SPDK threads per vhost controller. 2209 2210### vmd 2211 2212A new function, `spdk_vmd_fini`, has been added. It releases all resources acquired by the VMD 2213library through the `spdk_vmd_init` call. 2214 2215## v20.01: Optimized thin provisioning, FTL bdev, VMD hot plug, FUSED support 2216 2217### bdev 2218 2219A new function, `spdk_bdev_set_timeout`, has been added to set per descriptor I/O timeouts. 2220 2221A new class of functions `spdk_bdev_compare*`, have been added to allow native bdev support 2222of block comparisons and compare-and-write. 2223 2224A new class of bdev events, `SPDK_BDEV_EVENT_MEDIA_MANAGEMENT`, has been added to allow bdevs 2225which expose raw media to alert all I/O channels of pending media management events. 2226 2227A new API was added `spdk_bdev_io_get_aux_buf` allowing the caller to request 2228an auxiliary buffer for its own private use. The API is used in the same manner that 2229`spdk_bdev_io_get_buf` is used and the length of the buffer is always the same as the 2230bdev_io primary buffer. 'spdk_bdev_io_put_aux_buf' frees the allocated auxiliary 2231buffer. 2232 2233### blobfs 2234 2235Added boolean return value for function spdk_fs_set_cache_size to indicate its operation result. 2236 2237Added `blobfs_set_cache_size` RPC method to set cache size for blobstore filesystem. 2238 2239### blobstore 2240 2241Added new `use_extent_table` option to `spdk_blob_opts` for creating blobs with Extent Table descriptor. 2242Using this metadata format, dramatically decreases number of writes required to persist each cluster allocation 2243for thin provisioned blobs. Extent Table descriptor is enabled by default. 2244See the [Blobstore Programmer's Guide](https://spdk.io/doc/blob.html#blob_pg_cluster_layout) for more details. 2245 2246### dpdk 2247 2248Updated DPDK submodule to DPDK 19.11. 2249 2250### env_dpdk 2251 2252`spdk_env_dpdk_post_init` now takes a boolean, `legacy_mem`, as an argument. 2253 2254A new function, `spdk_env_dpdk_dump_mem_stats`, prints information about the memory consumed by DPDK to a file specified by 2255the user. A new utility, `scripts/dpdk_mem_info.py`, wraps this function and prints the output in an easy to read way. 2256 2257### event 2258 2259The functions `spdk_reactor_enable_framework_monitor_context_switch()` and 2260`spdk_reactor_framework_monitor_context_switch_enabled()` have been changed to 2261`spdk_framework_enable_context_switch_monitor()` and 2262`spdk_framework_context_switch_monitor_enabled()`, respectively. 2263 2264### ftl 2265 2266All NVMe dependencies were removed from ftl library. 2267From now ftl library is using bdev_zone API. 2268bdev_ftl becomes virtual bdev. 2269 2270`ctrlr` and `trid` fields in `spdk_ftl_dev_init_opts` structure were replaced by 2271`base_bdev_desc`. 2272 2273`bdev_ftl_create` RPC method `trtype` and `traddr` parameters were replaced by `base_bdev` 2274parameter. 2275 2276`spdk_ftl_module_init` and `spdk_ftl_module_fini` functions were removed. 2277 2278`spdk_ftl_punit_range` and `ftl_module_init_opts` structures were removed. 2279 2280### isa-l 2281 2282Updated ISA-L submodule to commit f3993f5c0b6911 which includes implementation and 2283optimization for aarch64. 2284 2285Enabled ISA-L on aarch64 by default in addition to x86. 2286 2287### nvme 2288 2289`delayed_pcie_doorbell` parameter in `spdk_nvme_io_qpair_opts` was renamed to `delay_cmd_submit` 2290to allow reuse in other transports. 2291 2292Added RDMA WR batching to NVMf RDMA initiator. Send and receive WRs are chained together 2293and posted with a single call to ibv_post_send(receive) in the next call to qpair completion 2294processing function. Batching is controlled by 'delay_cmd_submit' qpair option. 2295 2296The NVMe-oF initiator now supports plugging out of tree NVMe-oF transports. In order 2297to facilitate this feature, several small API changes have been made: 2298 2299The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 2300A new function, `spdk_nvme_transport_available_by_name`, has been added. 2301A function table, `spdk_nvme_transport_ops`, and macro, `SPDK_NVME_TRANSPORT_REGISTER`, have been added which 2302enable registering out of tree transports. 2303 2304A new function, `spdk_nvme_ns_supports_compare`, allows a user to check whether a given namespace supports the compare 2305operation. 2306 2307A new family of functions, `spdk_nvme_ns_compare*`, give the user access to submitting compare commands to NVMe namespaces. 2308 2309A new function, `spdk_nvme_ctrlr_cmd_get_log_page_ext`, gives users more granular control over the command dwords sent in 2310log page requests. 2311 2312### nvmf 2313 2314Add SockPriority option in [Transport] section, this can be used for NVMe-oF target 2315on TCP transport to set sock priority for the incoming TCP connections. 2316 2317The NVMe-oF target now supports plugging out of tree NVMe-oF transports. In order 2318to facilitate this feature, several small API changes have been made: 2319 2320The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 2321`spdk_nvmf_tgt_get_transport`, `spdk_nvmf_transport_opts_init`, and `spdk_nvmf_transport_create` all have been 2322modified to take a string. 2323A function table, `spdk_nvmf_transport_ops`, and macro, `SPDK_NVMF_TRANSPORT_REGISTER`, have been added which 2324enable registering out of tree transports. 2325 2326Add `spdk_nvmf_tgt_stop_listen()` that can be used to stop listening for 2327incoming connections for specified target and trid. Listener is not stopped 2328implicitly upon destruction of a subsystem any more. 2329 2330A custom NVMe admin command handler has been added which allows the user to use the real drive 2331attributes from one of the target NVMe drives when reporting drive attributes to the initiator. 2332This handler can be enabled via the `nvmf_set_config` RPC. 2333Note: In a future version of SPDK, this handler will be enabled by default. 2334 2335The SPDK target and initiator both now include compare-and-write functionality with one caveat. If using the RDMA transport, 2336the target expects the initiator to send both the compare command and write command either with, or without inline data. The 2337SPDK initiator currently respects this requirement, but this note is included as a flag for other initiators attempting 2338compatibility with this version of SPDK. 2339 2340The `spdk_nvme_ctrlr_opts` struct has been extended with new field `transport_ack_timeout` which allows 2341to configure transport ACK timeout. Applicable for RDMA transport only. 2342 2343### rpc 2344 2345A new RPC, `bdev_zone_block_create`, enables creating an emulated zoned bdev on top of a standard block device. 2346 2347A new RPC, `bdev_ocssd_create`, enables creating an emulated zoned bdev on top of an Open Channel SSD. 2348 2349A new RPC, `blobfs_set_cache_size`, enables managing blobfs cache size. 2350 2351A new RPC, `env_dpdk_get_mem_stats`, has been added to facilitate reading DPDK related memory 2352consumption stats. Please see the env_dpdk section above for more details. 2353 2354A new RPC, `framework_get_reactors`, has been added to retrieve a list of all reactors. 2355 2356`bdev_ftl_create` now takes a `base_bdev` argument in lieu of `trtype`, `traddr`, and `punits`. 2357 2358`bdev_nvme_set_options` now allows users to disable I/O submission batching with the `-d` flag 2359 2360`bdev_nvme_cuse_register` now accepts a `name` parameter. 2361 2362`bdev_uring_create` now takes arguments for `bdev_name` and `block_size` 2363 2364`nvmf_set_config` now takes an argument to enable passthru of identify commands to base NVMe devices. 2365Please see the nvmf section above for more details. 2366 2367### scsi 2368 2369`spdk_scsi_lun_get_dif_ctx` now takes an additional argument of type `spdk_scsi_task`. 2370 2371### sock 2372 2373Added spdk_sock_writev_async for performing asynchronous writes to sockets. This call will 2374never return EAGAIN, instead queueing internally until the data has all been sent. This can 2375simplify many code flows that create pollers to continue attempting to flush writes 2376on sockets. 2377 2378Added `impl_name` parameter in spdk_sock_listen and spdk_sock_connect functions. Users may now 2379specify the sock layer implementation they'd prefer to use. Valid implementations are currently 2380"vpp" and "posix" and NULL, where NULL results in the previous behavior of the functions. 2381 2382### thread 2383 2384`spdk_thread_send_msg` now returns int indicating if the message was successfully 2385sent. 2386 2387A new function `spdk_thread_send_critical_msg`, has been added to support sending a single message from 2388a context that may be interrupted, e.g. a signal handler. 2389 2390Two new functions, `spdk_poller_pause`, and `spdk_poller_resume`, have been added to give greater control 2391of pollers to the application owner. 2392 2393### util 2394 2395`spdk_pipe`, a new utility for buffering data from sockets or files for parsing 2396has been added. The public API is available at `include/spdk/pipe.h`. 2397 2398### bdev 2399 2400Added spdk_bdev_io_get_nvme_fused_status function for translating bdev_io status to NVMe status 2401code for fused compare-and-write operation. 2402 2403Added spdk_bdev_get_acwu function for getting block device atomic compare and write unit size. 2404 2405## v19.10: Zoned bdev API, Opal bdev, NVMe character devices 2406 2407### rpc 2408 2409Many of SPDK's RPCs were renamed to be more consistent and intuitive in this release. 2410The old names will continue to function, but will display a deprecation warning. 2411 2412Added optional parameters '--arbitration-burst' and '--low/medium/high-priority-weight' to 2413'bdev_nvme_set_options' RPC method. 2414 2415Added optional parameter '--md-size' to 'construct_null_bdev' RPC method. 2416 2417Added optional parameters '--dif-type' and '--dif-is-head-of-md' to 'construct_null_bdev' 2418RPC method. 2419 2420Added `blobfs_detect` RPC method to detect whether a blobfs exists on given bdev. 2421 2422Added `blobfs_create` RPC method to build blobfs on given bdev. 2423 2424Added `blobfs_mount` RPC method to mount blobfs on given bdev to a host path by FUSE. 2425Then on the host path, user can directly do some file operations which will be mapped 2426to blobfs. 2427 2428### bdev 2429 2430Added new parameter `cdw0` to `spdk_bdev_io_complete_nvme_status()` and 2431`spdk_bdev_io_get_nvme_status()` that allows setting/getting 2432the NVMe completion queue DW0 entry. This allows vendor specific IO commands 2433to return command specific completion info back to the initiator. 2434 2435Added `spdk_bdev_get_write_unit_size()` function for retrieving required number 2436of logical blocks for write operation. 2437 2438New zone-related fields were added to the result of the `get_bdevs` RPC call: 2439 2440- `zoned`: indicates whether the device is zoned or a regular 2441 block device 2442- `zone_size`: number of blocks in a single zone 2443- `max_open_zones`: maximum number of open zones 2444- `optimal_open_zones`: optimal number of open zones 2445 2446The `zoned` field is a boolean and is always present, while the rest is only available for zoned 2447bdevs. 2448 2449A new `spdk_bdev_open_ext` function has been added and `spdk_bdev_open` function has been deprecated. 2450The new open function introduces requirement to provide callback function that will be called by 2451asynchronous event such as bdev removal. `spdk_bdev_open_ext` function takes bdev name as 2452an argument instead of bdev structure to avoid a race condition that can happen when the bdev 2453is being removed between a call to get its structure based on a name and actually opening it. 2454 2455New 'resize' event has been added to notify about change of block count property of block device. 2456Event is delivered only if block device was opened with `spdk_bdev_open_ext` function. 2457 2458### bdev zone 2459 2460Added new public header for zoned bdev. Zoned bdev is an extension 2461of the bdev interface. 2462 2463`spdk_bdev_get_zone_size()`, `spdk_bdev_get_max_open_zones()`, `spdk_bdev_get_optimal_open_zones()` 2464APIs were added for retrieving zoned device information. 2465`spdk_bdev_get_zone_info()` API was added for retrieving information about zones in zoned 2466device. 2467Added `spdk_bdev_zone_management()` API for changing zone state. 2468`spdk_bdev_zone_append()` and `spdk_bdev_zone_append_with_md()` APIs were added for 2469appending data to a zone. 2470Added `spdk_bdev_io_get_append location()` function for retrieving append location for I/O. 2471Added `spdk_bdev_is_zoned()` function for checking if bdev supports zoned namespace semantics. 2472 2473### bdev opal 2474 2475EXPERIMENTAL: A new opal bdev has been added to support management of 2476NVMe self-encrypting drives through the Opal specification. Users can 2477create opal bdevs from an NVMe namespace bdev, if the controller 2478containing that namespace supports Opal. Currently this is only 2479supported for namespace ID=1. The following RPCs have been added to 2480support Opal: `bdev_nvme_opal_init`, `bdev_nvme_opal_revert`, 2481`bdev_opal_create`, `bdev_opal_delete`, `bdev_opal_get_info`, 2482`bdev_opal_new_user`, `bdev_opal_set_lock_state`. 2483It does not yet support recreating the opal bdevs after application restart. 2484This bdev module should be considered very experimental, and the RPCs may 2485change significantly in future releases. 2486 2487### delay bdev 2488 2489The `bdev_delay_update_latency` has been added to allow users to update 2490a latency value for a given delay bdev. 2491 2492### compress bdev 2493 2494A new RPC `bdev_compress_get_orphans` has been added to list compress bdevs 2495that were not loaded due to a missing pm metadata file. In this state they 2496can only be deleted. 2497 2498### null bdev 2499 2500Metadata support has been added to Null bdev module. 2501 2502Protection information support has been added to Null bdev module. 2503 2504### nvme 2505 2506Added `no_shn_notification` to NVMe controller initialization options, users can enable 2507it for NVMe controllers. When the option is enabled, the controller will not do the 2508shutdown process and just disable the controller, users can start their application 2509later again to initialize the controller to the ready state. 2510 2511A controller flag `SPDK_NVME_CTRLR_WRR_SUPPORTED` was added to indicate the controller 2512can support weighted round robin arbitration feature with submission queue. 2513 2514Added `arbitration_burst` option for arbitration feature, and added three 2515`low/medium/high_priority_weight` options for weighted round robin arbitration. 2516 2517Added `spdk_nvme_ns_cmd_write_uncorrectable`. 2518 2519Added new error handling and reporting functionality. This includes several 2520new API functions to facilitate applications recovering when a qpair or 2521controller fails. 2522 2523`spdk_nvme_ctrlr_reconnect_io_qpair` - Reconnects a failed I/O qpair. 2524`spdk_nvme_ctrlr_set_trid` - Sets the trid of an existing controller. Can be used to 2525change the trid for failover cases. 2526`spdk_nvme_ctrlr_is_failed` - Returns the failed state of a controller. 2527`spdk_nvme_ctrlr_fail` - Forces a controller into a failed state. 2528 2529Modified the return behavior of several API functions to better indicate to 2530applications when a qpair is failed. This list of functions includes: 2531 2532`spdk_nvme_qpair_process_completions` 2533`spdk_nvme_ns_cmd_*` 2534`spdk_nvme_ctrlr_process_admin_completions` 2535`spdk_nvme_ctrlr_cmd_*` 2536 2537These functions now return -ENXIO when the qpair or controller on which they 2538operate is failed. 2539 2540EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux 2541kernel for controller as well as for namespace and process ioctl requests as usual 2542from linux environment. 2543 2544### nvmf 2545 2546The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts` 2547as its only parameter. This new structure contains the max_subsystems parameter previously 2548passed into that function. 2549 2550A new public API function `spdk_nvmf_get_tgt` has been added which allows users to 2551retrieve a pointer to an `spdk_nvmf_tgt` object by supplying its name. In the special 2552case where an RPC or application only creates a single target, this function can accept 2553a null name parameter and will return the only available target. 2554 2555The majority of the NVMe-oF RPCs now accept an optional tgt_name parameter. This will 2556allow those RPCs to work with applications that create more than one target. 2557 2558Three new NVMe-oF RPCs have been added `nvmf_create_target`, `nvmf_delete_target`, and 2559`nvmf_get_targets`. These new RPCs provide a basic interface for managing multiple target 2560objects. In SPDK the target object defines a unique discovery service. As of this release, 2561these RPCs are not intended to be used with the in-tree SPDK target applications, spdk_tgt and 2562nvmf_tgt, which use a single, global target structure. As such, they are not included in scripts/rpc.py 2563 2564Three new header functions have also been added to help deal with multiple targets. 2565`spdk_nvmf_tgt_get_name` takes a target pointer as an argument and returns its human readable name. 2566`spdk_nvmf_get_first_target` takes no arguments and returns the first target in the global list. 2567`spdk_nvmf_get_next_tgt` takes a target pointer as an argument and returns the next one in the global list. 2568 2569The `spdk_nvmf_tgt_accept` takes additional argument allowing to pass arbitrary context 2570information to the `new_qpair` callback. This will simplify the code when having multiple 2571nvmf targets or when retrieving the context information from globals is not suitable. 2572 2573### blobstore 2574 2575A new `spdk_bdev_create_bs_dev_from_desc` function has been added and `spdk_bdev_create_bs_dev` 2576function has been deprecated. 2577The new create function can cowork with `spdk_bdev_open_ext` function, which provides callback 2578function that will be called by asynchronous event such as bdev removal. 2579 2580### blobfs_bdev 2581 2582A new blobfs module `bdev` has been added to simplify the operations of blobfs on bdev. 2583 2584Function `spdk_blobfs_bdev_detect` is added to detect whether blobfs exists on the given block device. 2585 2586Function `spdk_blobfs_bdev_create` is added to create a blobfs on the given block device. 2587 2588Function `spdk_blobfs_bdev_mount` is added to mount a blobfs on the given block device to 2589a host path by FUSE. Then, a new thread is created dedicatedly for one mountpoint to handle 2590FUSE request by blobfs API. 2591 2592### build 2593 2594Option to build FUSE components into blobfs_bdev module for mounting a blobfs filesystem. 2595It requires the installation of libfuse3. By default, it is disabled. And it will be 2596enabled if run `./configure` with `--with-fuse` option. 2597 2598### iSCSI 2599 2600Portals may no longer be associated with a cpumask. The scheduling of 2601connections is moving to a more dynamic model. 2602 2603An new RPC `iscsi_portal_group_set_auth` has been added to set CHAP authentication 2604for discovery sessions specific for the existing iSCSI portal group. This RPC overwrites 2605the setting by the global parameters for the iSCSI portal group. 2606 2607### socket 2608 2609Added `spdk_sock_is_connected` to check whether the socket is currently connected. 2610`spdk_sock_group_poll` now returns number of events on success. 2611 2612### env 2613 2614Added `spdk_pci_device_unclaim()` function to cleanup pci claim file. 2615 2616### event 2617 2618`framework_start_init` RPC no longer stops the application on error during 2619initialization. 2620 2621### DPDK 2622 2623Updated DPDK submodule to DPDK 19.08. 2624 2625### ocf 2626 2627Updated OCF submodule to OCF v19.06 2628 2629Along with update, new cache mode 'write only' was added. 2630 2631New cache modes added to use via RPC, wi - write invalidate and wa - write around. 2632 2633New version of OCF provides fully asynchronous management API. 2634 2635## v19.07: NVMe-oF FC Transport, VMD, NVMe-oF Persistent reservations, Bdev I/O with separate metadata 2636 2637### ftl 2638 2639EXPERIMENTAL: Added ability to mirror writes to persistent write buffer cache 2640to allow for recovery from dirty shutdown event. 2641 2642Added handling of Asynchronous Nand Management Events (ANM). 2643 2644### vmd 2645 2646EXPERIMENTAL: Added Intel Volume Management Device (VMD) driver. VMD is an integrated 2647controller inside the CPU PCIe root complex. It enables virtual HBAs for the connected 2648NVMe SSDs. `spdk_vmd_init()` enumerates NVMe SSDs behind VMD device and hook them into 2649SPDK PCI subsystem. `spdk_nvme_probe()` or `spdk_nvme_connect()` can be used to connect 2650NVMe driver to the device located at the given transport ID. 2651 2652To obtain transport ID of NVMe SSD behind VMD `spdk_lspci` can be used. 2653 2654Current implementation does not support hotplug. 2655 2656### blobfs 2657 2658Blobfs file asynchronous operations were added to public APIs. 2659 2660### util 2661 2662A new file API `spdk_posix_file_load` was added to load file content into a data buffer. 2663 2664New APIs `spdk_dif_ctx_set_data_offset`, `spdk_dif_verify_stream`, 2665`spdk_dif_get_range_with_md`, `spdk_dif_get_length_with_md` have been added, 2666and existing APIs `spdk_dif_ctx_init`, `spdk_dif_set_md_interleave_iovs`, and 2667`spdk_dif_generate_stream` have been refined to insert or strip DIF into iovs 2668windows of any alignment. 2669 2670New APIs `spdk_dif_ctx_set_remapped_init_ref_tag`, `spdk_dif_remap_ref_tag`, 2671and `spdk_dix_remap_ref_tag` have been added to remap DIF reference tag. 2672 2673New APIs `spdk_dif_update_crc32c` and `spdk_dif_update_crc32c_stream` have been 2674added to compute CRC-32C checksum for extended LBA payload. 2675 2676Bdevperf and bdevio applications now support starting tests with application specific 2677RPCs. Please see helper Python scripts in their respective directories. 2678This is a move towards simpler RPC-only configuration for all main 2679and auxiliary applications. 2680 2681Legacy INI style configuration for SPDK applications will become deprecated in SPDK 19.10, 2682and removed in SPDK 20.01. Please consider moving to JSON-RPC configuration files and/or 2683RPC driven run-time configuration. 2684 2685### nvmf 2686 2687EXPERIMENTAL: A Fibre Channel transport that supports Broadcom HBAs has been 2688added. This depends on the FC HBA driver at 2689https://github.com/ecdufcdrvr/bcmufctdrvr. See [the documentation](https://spdk.io/doc/nvmf.html#nvmf_fc_transport) 2690for more information. 2691 2692Persistent reservation emulation has been added to the NVMe-oF target. Persistent reservation 2693state is stored in a JSON file on the local filesystem between target restart. To support this, 2694an optional parameter to the RPC method `nvmf_subsystem_add_ns` called `--ptpl-file` was added. 2695This allows the user to specify which file to store the persistent reservation state in. Note 2696that this is done per namespace. 2697 2698The c2h success optimization under which a command capsule response is not sent 2699for reads is turned on by default. A config knob was added to allow disabling 2700the optimization. This will mostly be used for integration testing with 5.0.x kernels 2701while some compatibility fixes make their way down the pipeline for 5.1.x kernels. 2702 2703The sock priority setting of the TCP connection owned by the tcp transport is added. It is 2704used to optimize the TCP connection performance under designated traffic classes. And the 2705priority is used to differentiate the sock priority between SPDK NVMe-oF TCP target application 2706and other TCP based applications. 2707 2708Shared receive queue can now be disabled even for NICs that support it using the 2709`nvmf_create_transport` RPC method parameter `no_srq`. The actual use of a shared 2710receive queue is predicated on hardware support when this flag is not used. 2711 2712spdk_nvmf_get_optimal_poll_group was added, which is used to return the optimal 2713poll group for the qpair. And `ConnectionScheduler` configuration is added into the 2714[Nvmf] section in etc/spdk/nvmf.conf.in to demonstrate how to configure the connection 2715scheduling strategy among different spdk threads. 2716 2717Added infrastructure to retrieve global and per poll group NVMf statistics. 2718 2719DIF strip and insert is now supported for TCP transport. When it is enabled, DIF 2720setting is not exposed to the NVMe-oF initiator, and DIF is attached into data 2721for write I/O and stripped from data for read I/O. 2722 2723Added a field `dif_insert_or_strip` to struct spdk_nvmf_transport_opts, and 2724updated the related rpc function nvmf_create_transport to make this 2725configurable parameter available to users. The `dif_insert_or_strip` is relevant 2726for TCP transport for now and used to configure the DIF strip and insert. 2727 2728Added infrastructure to retrieve NVMf transport statistics. 2729 2730### notify 2731 2732The function `spdk_notify_get_types()` and `spdk_notify_get_events()` were 2733renamed to `spdk_notify_foreach_type()` and `spdk_notify_foreach_event()`, 2734respectively. And update type name of callback accordingly. 2735 2736### bdev 2737 2738The format of the data returned by the get_bdevs_iostat RPC has changed to 2739make it easier to parse. It now returns an object with a "ticks" object 2740and "bdevs" array with the per-bdev statistics. 2741 2742A new bdev module `delay` has been added which simulates a drive latency when placed 2743on top of a Null bdev. This module is intended only for testing and can be created using 2744the new RPC `bdev_delay_create`. That RPC takes the name of the underlying bdev as well 2745as average and p99 latency arguments for both read and write operations. Average latency is 2746defined as a value close to what you would expect a perf tool such as FIO to report back as 2747the mean latency of all I/O submitted to the drive. p99 latency is defined as the value one 2748would expect the drive to see the slowest 1% of I/O report. For underlying drives with already 2749significant latency, the latency values provided to the drive will be additive. This should be 2750taken into account if trying to achieve an artificial latency on top of an nvme drive or aio device. 2751 2752DIF reference tag remapping is now supported for partition type virtual bdev 2753modules. When using partition type virtual bdevs, block address space is 2754remapped during I/O processing and DIF reference tag is remapped accordingly. 2755 2756Added spdk_bdev_*_with_md() functions allowing for IO with metadata being transferred in 2757separate buffer. To check support for separate metadata, use spdk_bdev_is_md_separate(). 2758 2759All bdevs now have a UUID. For devices whose backing hardware does not provide a UUID, 2760one is automatically generated. Across runs of SPDK, bdevs whose UUID is automatically 2761generated may change. 2762 2763A new virtual bdev module `compress` has been added to provide compression services on top of 2764a thinly provisioned logical volume. See documentation for complete details. 2765 2766### nvme 2767 2768Added an optional parameter `--io-queue-requests` to RPC `set_bdev_nvme_options`, which 2769can be used to change the number of requests allocated for one NVMe I/O queue. For 2770very big I/O size, e.g. 128MiB, with this option user will not get an error due to 2771limited requests in NVMe driver layer. 2772 2773Added spdk_nvme_ctrlr_get_transport_id() to get the transport ID from a 2774previously attached controller. 2775 2776Nvme Opal library spdk_opal_cmd deprecated. Adding separate command APIs. 2777NVMe Opal library add support for activating locking SP which will make the transaction 2778from "Manufactured-Inactive" state to "Manufactured" state. Upon successfully invoking 2779of this method, lock and unlock features will be enabled. 2780NVMe Opal library add support for locking/unlocking range and list locking range info. 2781NVMe opal library add support for multiuser. Admin can enable user and add user to specific 2782locking range and the user can lock/unlock his range. 2783 2784Added spdk_nvme_ctrlr_io_cmd_raw_no_payload_build() allowing a caller to pass 2785a completely formed command to an NVMe submission queue (buffer addresses and all). 2786This is supported on the PCIe transport only. 2787 2788Added spdk_nvme_get_ctrlr_registers() to return a pointer to the virtual address 2789of the NVMe controller registers. This is supported on the PCIe transport only. 2790 2791Added additional options to the spdk_nvme_ctrlr_alloc_qpair() option parameter 2792structure to allow caller to override the virtual and optionally physical address 2793of the submission and completion queue pair to be created. This is supported on 2794the PCIe transport only. 2795 2796Added `disable_error_logging` to struct spdk_nvme_ctrlr_opts, which disables 2797logging of failed requests. By default logging is enabled. 2798 2799Added spdk_nvme_qpair_print_command(), spdk_nvme_qpair_print_completion() and 2800spdk_nvme_cpl_get_status_string(). Allowing for easier display of error messages. 2801 2802Added support for NVMe Sanitize command. 2803 2804### env 2805 2806The parameter `free_space` has been added to spdk_ring_enqueue() to wait when 2807the ring is almost full and resume when there is enough space available in 2808the ring. 2809 2810A new API `spdk_mempool_lookup` has been added to lookup the memory pool created 2811by the primary process. 2812 2813Added spdk_pci_get_first_device() and spdk_pci_get_next_device() to allow 2814iterating over PCI devices detected by SPDK. Because of this, all SPDK APIs 2815to attach/detach PCI devices are no longer thread safe. They are now meant to 2816be called from only a single thread only, the same only that called spdk_env_init(). 2817This applies to the newly added APIs as well. 2818 2819### vpp 2820 2821SPDK now supports VPP version 19.04.2, up from VPP 18.01. 2822 2823VPP socket abstraction now uses VPP Session API, instead of VLC (VPP Communications Library). 2824This allows for better control over sessions and queues. 2825Please see VPP documentation for more details: 2826[VPP Host Stack](https://wiki.fd.io/view/VPP/HostStack) 2827 2828### sock 2829 2830Add spdk_sock_get_optimal_sock_group(), which returns the optimal sock group for 2831this socket. When a socket is created, it is often assigned to a sock group using 2832spdk_sock_group_add_sock so that a set of sockets can be polled more efficiently. 2833For some network devices, it is optimal to assign particular sockets to specific 2834sock groups. This API is intended to provide the user with that information. 2835 2836spdk_sock_group_get_ctx() was added to return the context of the spdk_sock_group. 2837spdk_sock_group_create() is updated to allow input the user provided ctx. 2838spdk_sock_set_priority() is added to set the priority of the socket. 2839 2840### rpc 2841 2842Added thread_get_stats RPC method to retrieve existing statistics. 2843 2844Added nvmf_get_stats RPC method to retrieve NVMf subsystem statistics. 2845 2846Response buffers for RPC requests are now always pre-allocated, which implies 2847that all spdk_jsonrpc_begin_result() calls always succeed and return a valid 2848buffer for JSON response. RPC calls no longer need to check if the buffer is 2849non-NULL. 2850 2851Added SPDK_RPC_REGISTER_ALIAS_DEPRECATED to help with deprecation process when 2852renaming existing RPC. First time a deprecated alias is used, it will print 2853a warning message. 2854 2855RPC `get_rpc_methods` was renamed `rpc_get_methods`. The old name is still usable, 2856but is now deprecated. 2857 2858### blobstore 2859 2860A snapshot can now be deleted if there is only a single clone on top of it. 2861 2862### build 2863 2864Preliminary support for cross compilation is now available. Targeting an older 2865CPU on the same architecture using your native compiler can be accomplished by 2866using the `--target-arch` option to `configure` as follows: 2867 2868~~~bash 2869./configure --target-arch=broadwell 2870~~~ 2871 2872Additionally, some support for cross-compiling to other architectures has been 2873added via the `--cross-prefix` argument to `configure`. To cross-compile, set CC 2874and CXX to the cross compilers, then run configure as follows: 2875 2876~~~bash 2877./configure --target-arch=aarm64 --cross-prefix=aarch64-linux-gnu 2878~~~ 2879 2880### vhost 2881 2882A security vulnerability has been identified and fixed in SPDK Vhost-SCSI target. 2883A malicious client (e.g. a virtual machine) could send a carefully prepared, 2884invalid I/O request to crash the entire SPDK process. All users of SPDK Vhost-SCSI 2885target are strongly recommended to update. All SPDK versions < 19.07 are affected. 2886 2887By default, SPDK will now rely on upstream DPDK's rte_vhost instead of its fork 2888located inside SPDK repo. The internal fork is still kept around to support older 2889DPDK versions, but is considered legacy and will be eventually removed. 2890 2891`configure` will now automatically use the upstream rte_vhost if the used DPDK 2892version is >= 19.05. 2893 2894spdk_vhost_init() is now asynchronous and accepts a completion callback. 2895 2896### iscsi target 2897 2898A security vulnerability has been identified and fixed in SPDK iSCSI target. 2899A malicious client (e.g. an iSCSI initiator) could send a carefully prepared, 2900invalid I/O request to crash the entire SPDK process. All users of SPDK iSCSI 2901target are strongly recommended to update. All SPDK versions < 19.07 are affected. 2902 2903### thread 2904 2905Exposed spdk_set_thread() in order for applications to associate 2906with SPDK thread when necessary. 2907 2908Added spdk_thread_destroy() to allow framework polling the thread to 2909release resources associated with that thread. 2910 2911## v19.04: Compression bdev, Notification library, NVMe Opal support 2912 2913### nvme 2914 2915Added asynchronous probe support. New APIs spdk_nvme_probe_async(), 2916spdk_nvme_connect_async() and spdk_nvme_probe_poll_async() were added to enable 2917this feature. spdk_nvme_probe_async() and spdk_nvme_connect_async() return a 2918context associated with the specified controllers. Users then call 2919spdk_nvme_probe_poll_async() until it returns 0, indicating that the operation 2920completed. 2921 2922A new qpair creation option, delay_pcie_doorbell, was added. This can be passed 2923to spdk_nvme_alloc_io_qpair(). This makes the I/O submission functions, such as 2924spdk_nvme_ns_writev(), skip ringing the submission queue doorbell. Instead the 2925doorbell will be rung as necessary inside spdk_nvme_qpair_process_completions(). 2926This can result in significantly fewer MMIO writes to the doorbell register 2927under heavy load, greatly improving performance. 2928 2929spdk_nvme_ctrlr_get_regs_cmbsz() was added to report the size of the controller 2930memory buffer, if available. 2931 2932spdk_nvme_ctrlr_get_flags() was added to return controller feature 2933flags. Two flags are currently tracked: 2934SPDK_NVME_CTRLR_SGL_SUPPORTED 2935SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED 2936 2937The NVMe hotplug poller is now able to detach devices hot-removed from the system 2938via `/sys/bus/pci/devices/<bdf>/remove` and `/sys/bus/pci/devices/<bdf>/driver/unbind`. 2939 2940Opal support was added for scan, take ownership, revert TPer, and dumping device 2941info. The nvme_manage tool can be used to perform these operations. The public 2942API functions are spdk_nvme_ctrlr_security_receive() and 2943spdk_nvme_ctrlr_security_send(). This module should be considered experimental 2944pending additional features and tests. 2945 2946The NVMe-oF initiator is now able to transfer IO whose size is more than 128KiB 2947if the corresponding NVMe-oF target allows. 2948 2949### raid 2950 2951Added new strip_size_kb rpc param on create to replace the more ambiguous 2952strip_size. The strip_size rpc param is deprecated. 2953 2954Changed the raid bdev product_name from "Pooled Device" to "Raid Volume" 2955 2956### thread 2957 2958Added spdk_thread_has_pollers() function to verify if there are any registered 2959pollers to be run on the thread. Added spdk_thread_is_idle() function to check 2960if there are any scheduled operations to be performed on the thread at given 2961time. 2962 2963spdk_thread_create() now takes an optional CPU affinity mask that is passed to 2964the scheduler. 2965 2966spdk_thread_lib_init() now takes an optional context size. For each thread 2967created, an additional region of memory of this size will be allocated. A 2968pointer to this region of memory can be obtained by calling 2969spdk_thread_get_ctx(). The inverse operation is also available via 2970spdk_thread_get_from_ctx(). 2971 2972spdk_thread_poll() now optionally accepts the current time, in ticks. This can 2973be used to avoid some calls to spdk_get_ticks() internally. 2974 2975spdk_app_start() now only accepts a single context argument. 2976 2977### bdev 2978 2979An new API `spdk_bdev_get_data_block_size` has been added to get size of data 2980block except for metadata. 2981 2982spdk_vbdev_register() has been deprecated. spdk_bdev_register() should be used 2983instead. 2984 2985A mechanism for acquiring and releasing data buffers from bdev modules, used 2986to perform zero copy operations, was added. 2987 2988New APIs spdk_bdev_get_md_size(), spdk_bdev_is_md_interleaved(), spdk_bdev_get_dif_type(), 2989spdk_bdev_is_dif_head_of_md(), and spdk_bdev_is_dif_check_enabled() have been 2990added to get metadata and DIF settings. 2991 2992Bdevs claimed by the `examine_config` callback will be now further examined in the 2993`examine_disk` callback. 2994 2995spdk_bdev_io_get_io_channel() was added as a convenient way to get an io_channel 2996from a bdev_io. 2997 2998### ocf 2999 3000Updated OCF submodule to OCF v19.3.2 3001 3002Added support for many-to-one configuration for OCF bdev. 3003Multiple core devices can now be cached on single cache device. 3004 3005Added persistent metadata support, allowing to restore cache state after shutdown. 3006During start of SPDK application, the devices are examined and if OCF metadata 3007is present - appropriate OCF bdevs will be recreated. 3008 3009Added Write-Back mode support. In this mode, data is first written to 3010caching device and periodically synchronized to the core devices. 3011Dirty data is saved as persistent metadata on cache device, 3012allowing for safe restore during application restart. 3013For more details please see OCF documentation: 3014[OpenCAS cache configuration](https://open-cas.github.io/cache_configuration.html) 3015 3016### NVMe-oF Target 3017 3018Support for per-device shared receive queues in the RDMA transport has been added. 3019It is enabled by default for any device that supports it. 3020 3021The size of a shared receive queue is defined by transport configuration file parameter 3022`MaxSRQDepth` and `nvmf_create_transport` RPC method parameter `max_srq_depth`. 3023Default size is 4096. 3024 3025Add model number as parameter to construct_nvmf_subsystem (-d option), 3026rather than using hardcoded define. 3027 3028DIF passthrough feature has been added. DIF setting of the allocated bdevs is 3029exposed to the NVMe-oF initiator and data with DIF from the NVMe-oF initiator is 3030passed through to the allocated bdevs. 3031 3032### env 3033 3034The `phys_addr` parameter in spdk_malloc() and spdk_zmalloc() has been deprecated. 3035For retrieving physical addresses, spdk_vtophys() should be used instead. 3036 3037spdk_realloc() has been added to reallocate DMA/shared memory. 3038 3039spdk_pci_device_is_removed() has been added to let the upper-layer SPDK drivers know 3040that device has a pending external hotremove request. 3041 3042spdk_env_fini() and spdk_env_dpdk_post_fini() were added to release any resources 3043allocated by spdk_env_init() or spdk_env_dpdk_post_init() respectively. It is expected 3044that common usage of those functions is to call them just before terminating the process. 3045 3046Added spdk_iommu_is_enabled() to report if SPDK application is using IOMMU for DMA. 3047 3048### DPDK 3049 3050Dropped support for DPDK 17.07 and earlier, which SPDK won't even compile with right now. 3051 3052Updated DPDK submodule to DPDK 19.02. 3053 3054### rpc 3055 3056New `get_spdk_version` RPC method is introduced to get version info of the running SPDK application. 3057 3058The `start_nbd_disk` RPC method now take nbd_device as an optional parameter. If nbd_device 3059is specified, use that specified nbd device. If it's not specified, pick available one. 3060 3061### iSCSI target 3062 3063DIF strip and insert is now supported. DIF settings are not exposed to the iSCSI initiator. 3064DIF is attached into data for write I/O and stripped from data for read I/O. 3065 3066### vhost 3067 3068Added experimental support for running with the external, upstream rte_vhost library. 3069This can be enabled by configuring SPDK with an `--without-internal-vhost-lib` flag. 3070The minimum supported rte_vhost version (DPDK version) is 19.05-rc1. 3071 3072As a result of fuzz testing, a lot of data races in vhost-scsi LUN hotplug path were identified and 3073fixed. Those data races could have potentially resulted in SPDK crashes, RPC hangs, or memory leaks 3074if Vhost-SCSI LUN hotplug RPCs were executed while connected VMs were in the middle of restarting. 3075 3076The SCSI target id in `add_vhost_scsi_lun` RPC is now optional. If `-1` is passed, the first 3077unoccupied target id will be used. 3078 3079### AIO 3080 3081AIO bdev module can now reap I/O completions directly from userspace, significantly improving 3082the overall performance. 3083 3084### blobfs 3085 3086Synchronous IO operations no longer use spdk_io_channel, but instead use 3087spdk_fs_thread_ctx. The behavior is otherwise identical. 3088 3089### OCF 3090 3091Added support for caching multiple bdevs using a single bdev as a cache. 3092 3093### notify 3094 3095Added the notify library that provides a high performance local event bus 3096between libraries. Example usage was added to bdev module, which reports 3097notifications for added and removed bdevs via RPC. 3098 3099### sock 3100 3101Added new API spdk_sock_readv() to the sock library for performing vectored 3102reads. 3103 3104### event 3105 3106The function spdk_subsystem_init() no longer requires spdk_event as an argument. 3107 3108Changed API of spdk_subsystem_config_json() to no longer be asynchronous. 3109 3110### io_uring 3111 3112A bdev module that issues I/O to kernel block devices using the new io_uring Linux kernel 3113API was added. This module requires liburing. 3114 3115### build 3116 3117Options to easily compile with profile guided optimization have been added to 3118`configure`. To use profile guided optimization with SPDK, run 3119`./configure --with-pgo-capture`, build SPDK, then run a workload of your 3120choosing. Then, simply run `./configure --with-pgo-enable` and recompile to 3121build using the generated profile data. Profile guided optimization can yield 3122very large performance improvements, especially on GCC 8 and clang 7. This may 3123be combined with link time optimization which has been available under the 3124`--enable-lto` configure option for several releases. 3125 3126### compression bdev/reduce library 3127 3128Added "reduce" block compression scheme based on using SSDs for storing 3129compressed blocks of storage and persistent memory for metadata. Please see 3130[compression](https://spdk.io/doc/bdev.html) for more details. 3131 3132## v19.01: NVMe-oF TCP/IP Transport, Open Channel SSD Flash Translation Layer, Caching bdev based on OCF, ISA-L Support, DIF/DIX library 3133 3134### ocf bdev 3135 3136New virtual bdev module based on [Open CAS Framework](https://open-cas.github.io/) has been added. 3137This module allows for the use of one bdev to act as a high performance cache in front of another bdev. 3138Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_cas) for more details. 3139Only write through mode is currently supported and this feature is considered experimental. 3140 3141### event framework 3142 3143For `spdk_app_parse_args`, add return value to the callback which parses application 3144specific command line parameters to protect SPDK applications from crashing by invalid 3145values from user input. 3146 3147By default, all SPDK applications will now reserve all hugepages at runtime. The pre-reserved 3148memory size can be still set with `-s` or `--mem-size` option, although the default value 3149was reduced down to 0. 3150 3151A custom hugetlbfs directory can now be specified via spdk_app_opts. 3152This can be used to configure hugepages with different sizes, a different size limit, 3153or different access permissions than the system's default hugepage pool. 3154SPDK applications can specify a custom hugetlbfs mount with the `--huge-dir` option. 3155 3156### environment 3157 3158spdk_vtophys() has been refactored to accept length of the translated region as a new 3159parameter. The function will now update that parameter with the largest possible value 3160for which the memory is contiguous in the physical memory address space. 3161 3162The following functions were removed: 3163 3164- spdk_pci_nvme_device_attach() 3165- spdk_pci_nvme_enumerate() 3166- spdk_pci_ioat_device_attach() 3167- spdk_pci_ioat_enumerate() 3168- spdk_pci_virtio_device_attach() 3169- spdk_pci_virtio_enumerate() 3170 3171They were replaced with generic spdk_pci_device_attach() and spdk_pci_enumerate() which 3172require a new spdk_pci_driver object to be provided. It can be one of the following: 3173 3174- spdk_pci_nvme_get_driver() 3175- spdk_pci_ioat_get_driver() 3176- spdk_pci_virtio_get_driver() 3177 3178spdk_pci_hook_device() and spdk_pci_unhook_device() were added. Those allow adding a virtual 3179spdk_pci_device into the SPDK PCI subsystem. A virtual device calls provided callbacks for 3180each BAR mapping request or PCI config access. It's attachable with spdk_pci_device_attach() 3181or spdk_pci_enumerate() like any other device. 3182 3183A new spdk_pause() function was added to pause CPU execution for an implementation specific 3184amount of time. Quoting from DPDK function this is based on: "This call is intended for 3185tight loops which poll a shared resource or wait for an event. A short pause within the loop 3186may reduce the power consumption." 3187 3188A new public header file env_dpdk.h has been introduced, and function spdk_env_dpdk_post_init 3189is added into it. If user is using DPDK, and already called rte_eal_init, then include 3190include/spdk/env_dpdk.h, and call spdk_env_dpdk_post_init() instead of spdk_env_init. 3191 3192ISA-L has been added as an SPDK submodule. ISA-L is enabled by default on x86 architecture 3193to accelerate algorithms such as CRC for iSCSI and NVMe-oF. Users may still disable ISA-L 3194by explicitly passing --without-isal to the configure script. 3195 3196### util 3197 3198A new uuid API `spdk_uuid_copy` was added to make a copy of the source uuid. 3199 3200An new parameter `init_crc` representing the initial CRC value was added to 3201`spdk_crc16_t10dif`. The parameter can be used to calculate a CRC value spanning 3202multiple separate buffers. 3203 3204New DIF APIs were added to generate and verify DIF by byte granularity for both DIF and DIX 3205formats. Among them, DIF with copy APIs will be usable to emulate DIF operations such as DIF 3206insert and strip. 3207 3208Added `spdk_strtol` and `spdk_strtoll` to provide additional error checking around `strtol` 3209and `strtoll`. 3210 3211Added `spdk_sprintf_append_realloc` and `spdk_vsprintf_append_realloc` for appending a string 3212with automatic buffer re-allocation. 3213 3214### nvme 3215 3216Wrapper functions spdk_nvme_ctrlr_security_send() and spdk_nvme_ctrlr_security_receive() are 3217introduced to support further security protocol development. 3218 3219admin_timeout_ms was added to NVMe controller initialization options, users 3220can change the default value when probing a controller. 3221 3222Add two new fields "header_digest" and "data_digest" in struct spdk_nvme_ctrlr_opts, 3223it will be used to enable the digest support for the NVMe/TCP transport. 3224 3225Add a new TCP/IP transport(located in lib/nvme/nvme_tcp.c) in nvme driver. With 3226this new transport, it can be used to connect the NVMe-oF target with the 3227same TCP/IP support. 3228 3229Added API, spdk_nvme_ctrlr_is_discovery(), to indicate whether the ctrlr 3230arg refers to a Discovery Controller or not. 3231 3232Added an API function `spdk_nvme_host_id_parse` and corresponding object `spdk_nvme_host_id` 3233for parsing host address and host service ID arguments on a per connection basis. 3234 3235The RPC `construct_nvme_bdev` now allows a user to specify a source address and service id for the host to 3236use when connecting to the controller backing the NVMe bdev. 3237 3238### NVMe-oF Target 3239 3240The `spdk_nvmf_tgt_opts` struct has been deprecated in favor of `spdk_nvmf_transport_opts`. 3241Users will no longer be able to specify target wide I/O parameters. `spdk_nvmf_tgt_listen` 3242will also no longer implicitly initialize a transport with the default target options (since 3243there are none). Instead, a user must manually instantiate the transport with `spdk_nvmf_transport_create` 3244prior to calling `spdk_nvmf_tgt_listen`. 3245 3246Related to the previous change, the rpc `set_nvmf_target_options` has been renamed to 3247`set_nvmf_target_max_subsystems` to indicate that this is the only target option available for the user to edit. 3248 3249Added fields `num_shared_buffers` and `buf_cache_size` in struct spdk_nvmf_transport_opts, 3250and also updated the related rpc function nvmf_create_transport, to make this 3251configurable parameter available to users. The `num_shared_buffers` is used to 3252configure the shared buffer numbers of the transport used by RDMA or TCP transport. 3253`buf_cache_size` configures number of shared buffers to cache per poll group. 3254 3255### nvmf 3256 3257Add a new TCP/IP transport (located in lib/nvmf/tcp.c). With this transport, 3258the SPDK NVMe-oF target can have a new transport, and can serve the NVMe-oF 3259protocol via TCP/IP from the host. 3260 3261Added optional mechanism to modify the RDMA transport's behavior when creating protection domains and registering memory. 3262By default, the RDMA transport will use the ibverbs library to create protection domains and register memory. 3263Using `spdk_nvme_rdma_init_hooks` will subvert that and use an existing registration. 3264 3265### bdev 3266 3267Added `enable_bdev_histogram` and `get_bdev_histogram` RPC commands to allow gathering latency data for specified bdev. 3268Please see [documentation](https://spdk.io/doc/bdev.html#rpc_bdev_histogram) for more details. 3269 3270Added `required_alignment` field to `spdk_bdev`, that specifies an alignment requirement for data buffers associated with an spdk_bdev_io. 3271Bdev layer will automatically double buffer any spdk_bdev_io that violates this alignment, before the spdk_bdev_io is submitted to the bdev module. 3272 3273On shutdown, bdev unregister now proceeds in top-down fashion, with 3274claimed bdevs skipped (these will be unregistered later, when virtual 3275bdev built on top of the respective base bdev unclaims it). This 3276allows virtual bdevs to be shut down cleanly as opposed to the 3277previous behavior that didn't differentiate between hotremove and 3278planned shutdown. 3279 3280The `delete_bdev` RPC is now deprecated. Users should instead use the specific deletion RPC 3281for the bdev type to be removed (i.e. delete_malloc_bdev). 3282 3283Added support for separate bandwidth rate limits for read and write to QoS in bdev layer. 3284 3285Bdev I/O statistics now track unmap operations. 3286 3287### logical volumes 3288 3289Logical volume bdev can now be marked as read only using `set_read_only_lvol_bdev` RPC. 3290This allows for basing clones on top of lvol_bdev without first creating a snapshot. 3291 3292Added option to change method for data erasure when deleting lvol or resizing down. 3293Default of unmapping clusters can now be changed to writing zeroes or no operation. 3294 3295Added option to change method for erasing data region on lvol store creation. 3296Default of unmapping can now be changed to writing zeroes or no operation. 3297 3298### log 3299 3300"trace flags" are now referred to as "log flags" in the SPDK log API. The 3301set_trace_flag, clear_trace_flag and get_trace_flags RPCs are now deprecated, 3302and set_log_flag, clear_log_flag and get_log_flags RPCs have been added. 3303 3304### trace 3305 3306New `trace_record` application was added. It can be used to poll spdk trace shm file and 3307append any new trace entries into another specified file. This can help retain those entries 3308that would otherwise be overwritten in the shm file. See 3309[Capturing sufficient trace events](https://spdk.io/doc/nvmf_tgt_tracepoints.html#capture_trace_events) 3310for more details. 3311 3312Number of trace entries in circular buffer per lcore can now be assigned by starting SPDK app 3313with argument "--num-trace-entries <NUM>" provided. 3314 3315New `get_tpoint_group_mask` RPC was added to get current tpoint_group_mask, and 3316each tpoint group status. 3317New `enable_tpoint_group` and `disable_tpoint_group` RPC were added to enable or 3318disable a specific tpoint group. 3319 3320### ftl 3321 3322EXPERIMENTAL: Added basic flash translation layer module allowing for using Open Channel SSDs as 3323block devices. The module is split into the library (located in lib/ftl) and bdev_ftl 3324(lib/bdev/ftl). See the [documentation](https://spdk.io/doc/ftl.html) for more details. 3325 3326### vhost 3327 3328A security vulnerability has been identified and fixed in the SPDK vhost target. A malicious 3329vhost client (i.e. virtual machine) could carefully construct a circular descriptor chain which 3330would result in a partial denial of service in the SPDK vhost target. These types of descriptor 3331chains are now properly detected by the vhost target. All SPDK vhost users serving untrusted 3332vhost clients are strongly recommended to upgrade. (Reported by Dima Stepanov and Evgeny 3333Yakovlev.) 3334 3335Vhost SCSI and Vhost Block devices can now accept multiple connections on the same socket file. 3336Each connection (internally called a vhost session) will have access to the same storage, but 3337will use different virtqueues, different features and possibly different memory. 3338 3339### vhost scsi 3340 3341SCSI target hotremove can now be performed even without the VIRTIO_SCSI_F_HOTPLUG feature negotiated. 3342Regardless of VIRTIO_SCSI_F_HOTPLUG support, the hotremoval will be still reported through SCSI sense codes. 3343 3344### DPDK 3345 3346DPDK submodule was updated to DPDK 18.11. Note that SPDK does not fully leverage the new 3347multi-process device hotplug yet and continues to work the same way it always did. 3348 3349Dropped support for DPDK 16.07 and earlier, which SPDK won't even compile with right now. 3350 3351### RPC 3352 3353The following RPC commands deprecated in the previous release are now removed: 3354 3355- construct_virtio_user_scsi_bdev 3356- construct_virtio_pci_scsi_bdev 3357- construct_virtio_user_blk_bdev 3358- construct_virtio_pci_blk_bdev 3359- remove_virtio_scsi_bdev 3360- construct_nvmf_subsystem 3361 3362### Miscellaneous 3363 3364The configure options `--with-raid` and `--without-raid` that were deprecated in the previous 3365release are now removed. 3366 3367### nbd 3368 3369Starting nbd using `spdk_nbd_start` is now performed asynchronously. 3370 3371### net framework 3372 3373Net framework initialization and finish is now done asynchronously. 3374 3375### rpc 3376 3377Added `spdk_rpc_is_method_allowed` function for checking whether method is permitted in a given state. 3378Added `spdk_rpc_get_state` to check current state of RPC server. 3379RPC `wait_subsystem_init` has been added to allow clients to block until all subsystems are initialized. 3380 3381### json rpc 3382 3383JSON RPC client is now running in non-blocking mode. Requests are sent and received during spdk_jsonrpc_client_poll. 3384JSON RPC server can now receive a callback on connection termination or server shutdown using `spdk_jsonrpc_conn_add_close_cb` 3385and `spdk_jsonrpc_conn_del_close_cb`. 3386 3387## v18.10: Dynamic memory allocation, Crypto Virtual Bdev, jsonrpc-client, SPDKCLI iSCSI and NVMe-oF support 3388 3389### nvme 3390 3391spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive() 3392were added to support sending or receiving security protocol data to or from 3393nvme controller. 3394 3395spdk_nvme_ns_get_extended_sector_size() was added. This function includes 3396the metadata size per sector (if any). spdk_nvme_ns_get_sector_size() still 3397returns only the data size per sector, not including metadata. 3398 3399New `send_nvme_cmd` RPC was added to allow sending NVMe commands directly to NVMe controller. 3400See the [send_nvme_cmd](http://spdk.io/doc/jsonrpc.html#rpc_send_nvme_cmd) documentation 3401for more details. 3402 3403### Build System 3404 3405New `configure` options, `--with-shared` and `--without-shared` 3406[default], provide the capability to build, or not, SPDK shared libraries. 3407This includes the single SPDK shared lib encompassing all of the SPDK 3408static libs as well as individual SPDK shared libs corresponding to 3409each of the SPDK static ones. Although the production of the shared 3410libs conforms with conventional version naming practices, such naming 3411does not at this time confer any SPDK ABI compatibility claims. 3412 3413### bdev 3414 3415spdk_bdev_alias_del_all() was added to delete all alias from block device. 3416 3417A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev 3418Framework. The module initially uses a software AESNI CBC cipher with experimental support for the 3419Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work 3420may include additional ciphers as well as consideration for authentication. 3421 3422The RAID virtual bdev module is now always enabled by default. The configure --with-raid and 3423--without-raid options are now ignored and deprecated and will be removed in the next release. 3424 3425Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer. 3426See the new [set_bdev_qos_limit](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit) 3427documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at 342818.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both 3429bandwidth and IOPS limits. 3430 3431spdk_bdev_config_json() and corresponding `get_bdevs_config` RPC was removed. 3432 3433### Environment Abstraction Layer and Event Framework 3434 3435The size parameter of spdk_mem_map_translate is now a pointer. This allows the 3436function to report back the actual size of the translation relative to the original 3437request made by the user. 3438 3439A new structure spdk_mem_map_ops has been introduced to hold memory map related 3440callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc 3441in lieu of the notify callback. 3442 3443### DPDK 18.08 3444 3445The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically 3446utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1. 3447 3448Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup, 3449but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try 3450to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using 3451only as many hugepages as it is really needed. 3452 3453Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed 3454to be physically contiguous. 3455 3456### I/OAT 3457 3458I/OAT driver can now reinitialize I/OAT channels after encountering DMA errors. 3459 3460### iscsi target 3461 3462Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC 3463method for CHAP authentication in discovery sessions have been changed to 3464align with `construct_target_node` RPC method. Old names are still usable 3465but will be removed in future release. 3466 3467`set_iscsi_discovery_auth` and `set_iscsi_target_node_auth` RPC methods have 3468been added to set CHAP authentication for discovery sessions and existing 3469target nodes, respectively. 3470 3471The SPDK iSCSI target supports an AuthFile which can be used to load CHAP 3472shared secrets when the iSCSI target starts. SPDK previously provided a 3473default location for this file (`/usr/local/etc/spdk/auth.conf`) if none was 3474specified. This default has been removed. Users must now explicitly specify 3475the location of this file to load CHAP shared secrets from a file, or use 3476the related iSCSI RPC methods to add them at runtime. 3477 3478### iscsi initiator 3479 3480The SPDK iSCSI initiator is no longer considered experimental and becomes 3481a first-class citizen among bdev modules. The basic usage has been briefly 3482described in the bdev user guide: [iSCSI bdev](https://spdk.io/doc/bdev.html#bdev_config_iscsi) 3483 3484### Miscellaneous 3485 3486The SPDK application framework can now parse long name command line parameters. 3487Most single-character parameters have a long name equivalent now. See the 3488[Command Line Parameters](https://spdk.io/doc/app_overview.html) documentation 3489for details or use the `--help` command line parameter to list all available 3490params. 3491 3492bdevperf `-s` param (io size) was renamed to `-o` as `-s` had been already 3493used by existing apps for memory size. 3494 3495bdevio can now accept all SPDK command line parameters. The config now has to 3496be provided with `-c` or `--config` param. 3497 3498The following ioat/perf and nvme/perf parameters were renamed as well: 3499 `-s` (io size) to `-o` 3500 `-d` (mem size) to `-s` 3501 3502The ReactorMask config file parameter has been deprecated. Users should 3503use the -m or --cpumask command line option to specify the CPU core mask 3504for the application. 3505 3506Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt 3507and vhost. Config file pathnames may now only be specified using the 3508-c command line option. 3509 3510Users may no longer set DPDK_DIR in their environment to specify the 3511location of the DPDK installation used to build SPDK. Using DPDK_DIR 3512has not been the documented nor recommended way to specify the DPDK 3513location for several releases, but removing it ensures no unexpected 3514surprises for users who may have DPDK_DIR defined for other reasons. 3515Users should just use the "configure" script to specify the DPDK 3516location before building SPDK. 3517 3518Although we know that many developers still use Python 2 we are officially 3519switching to Python3 with requirement that all new code must be valid also 3520for Python 2 up to the EOL which is year 2020. 3521 3522Invoking interpreter explicitly is forbidden for executable scripts. There 3523is no need to use syntax like "python ./scripts/rpc.py". All executable 3524scripts must contain proper shebang pointing to the right interpreter. 3525Scripts without shebang mustn't be executable. 3526 3527A Python script has been added to enable conversion of old INI config file 3528to new JSON-RPC config file format. This script can be found at 3529scripts/config_converter.py. Example how this script can be used: 3530~~~{.sh} 3531cat old_format.ini | scripts/config_converter.py > new_json_format.json 3532~~~ 3533 3534### Sock 3535 3536Two additional parameters were added to spdk_sock_get_addr() for the server 3537port and client port. These parameters are named "sport" and "cport" 3538respectively. 3539 3540### Virtio 3541 3542The following RPC commands have been deprecated: 3543 3544- construct_virtio_user_scsi_bdev 3545- construct_virtio_pci_scsi_bdev 3546- construct_virtio_user_blk_bdev 3547- construct_virtio_pci_blk_bdev 3548- remove_virtio_scsi_bdev 3549 3550The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev` 3551command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev` 3552was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI 3553devices. 3554 3555### Blobfs 3556 3557spdk_file_get_id() returning unique ID for the file was added. 3558 3559### JSON 3560 3561Added jsonrpc-client C library intended for issuing RPC commands from applications. 3562 3563Added API enabling iteration over JSON object: 3564 3565- spdk_json_find() 3566- spdk_json_find_string() 3567- spdk_json_find_array() 3568- spdk_json_object_first() 3569- spdk_json_array_first() 3570- spdk_json_next() 3571 3572### Blobstore 3573 3574Blobstore I/O operations are now based on io_units, instead of blobstore page size. 3575The io_unit size is now the same as the underlying block device's block size. 3576Logical volumes built on a block device with 512B block size can now be used as boot devices 3577in QEMU. 3578 3579### SPDKCLI 3580 3581The SPDKCLI interactive command tool for managing SPDK is no longer considered experimental. 3582Support for the iSCSI and NVMe-oF targets has been added. 3583 3584## v18.07: Raid, Infrastructure Improvements, Bug Fixes 3585 3586### bdev 3587 3588A new public header file bdev_module.h has been introduced to facilitate the 3589development of new bdev modules. This header includes an interface for the 3590spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of 3591multiple virtual bdevs on top of a single base bdev and should act as the 3592primary API for module authors. 3593 3594spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide 3595options. 3596 3597A mechanism for handling out of memory condition errors (ENOMEM) returned from 3598I/O submission requests at the bdev layer has been added. See 3599spdk_bdev_queue_io_wait(). 3600 3601The spdk_bdev_get_io_stat() function now returns cumulative totals instead of 3602resetting on each call. This allows multiple callers to query I/O statistics 3603without conflicting with each other. Existing users will need to adjust their 3604code to record the previous I/O statistics to calculate the delta between calls. 3605 3606I/O queue depth tracking and samples options have been added. See 3607spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and 3608spdk_bdev_set_qd_sampling_period(). 3609 3610### RAID module 3611 3612A new bdev module called "raid" has been added as experimental module which 3613aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note 3614that vhost will not work with this module because it does not yet have support 3615for multi-element io vectors. 3616 3617### Log 3618 3619The debug log component flag available on several SPDK applications has been 3620renamed from `-t` to `-L` to prevent confusion with tracepoints and to allow the 3621option to be added to tools that already use `-t` to mean something else. 3622 3623### Blobstore 3624 3625A new function, spdk_bs_dump(), has been added that dumps all of the contents of 3626a blobstore to a file pointer. This includes the metadata and is very useful for 3627debugging. 3628 3629Two new operations have been added for thin-provisioned blobs. 3630spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions 3631of the blob and populate them with the correct data by reading from the backing 3632blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate 3633clusters that correspond to data in the blob's immediate parent. Clusters 3634allocated to grandparents or that aren't allocated at all will remain 3635thin-provisioned. 3636 3637### BlobFS 3638 3639Changed the return type of spdk_file_truncate() from void to int to allow the 3640propagation of `ENOMEM` errors. 3641 3642### NVMe Driver 3643 3644The new API functions spdk_nvme_qpair_add_cmd_error_injection() and 3645spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error 3646emulation. Users can set a specified command to fail with a particular error 3647status. 3648 3649Changed the name `timeout_sec` parameter to `timeout_us` in 3650spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from 3651uint32_t to uint64_t. This will give users more fine-grained control over the 3652timeout period. 3653 3654Basic support for Open Channel SSDs was added. See nvme_ocssd.h 3655 3656### NVMe Over Fabrics 3657 3658The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback 3659as a parameter. 3660 3661spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs. 3662 3663spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed 3664number of namespaces for a given subsystem. 3665 3666### Build System 3667 3668The build system now generates a combined shared library (libspdk.so) that may 3669be used in place of the individual static libraries (libspdk_*.a). The combined 3670library includes all components of SPDK and is intended to make linking against 3671SPDK easier. The static libraries are also still provided for users that prefer 3672to link only the minimal set of components required. 3673 3674### git pre-commit and pre-push hooks 3675 3676The pre-commit hook will run `scripts/check_format.sh` and verify there are no 3677formatting errors before allowing `git commit` to run. The pre-push hook runs 3678`make CONFIG_WERROR=y` with and without `CONFIG_DEBUG=y` using both the gcc and 3679clang compiler before allowing `git push` to run. Following each DEBUG build 3680`test/unit/unittest.sh` is run and verified. Results are recorded in the 3681`make.log` file. 3682 3683To enable type: 'git config core.hooksPath .githooks'. To override after 3684configuration use the `git --no-verify` flag. 3685 3686### RPC 3687 3688The `start_nbd_disk` RPC method now returns the path to the kernel NBD device node 3689rather than always returning `true`. 3690 3691### DPDK 18.05 3692 3693The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports 3694dynamic memory allocation, but due to some issues found after the DPDK 18.05 release, 3695that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use 3696the legacy memory allocation model. The plan is to enable dynamic memory allocation 3697after the DPDK 18.08 release which should fix these issues. 3698 3699### Environment Abstraction Layer and Event Framework 3700 3701The spdk_mem_map_translate() function now takes a size parameter to indicate the size of 3702the memory region. This can be used by environment implementations to validate the 3703requested translation. 3704 3705The I/O Channel implementation has been moved to its own library - lib/thread. The 3706public API that was previously in spdk/io_channel.h is now in spdk/thread.h The 3707file spdk/io_channel.h remains and includes spdk/thread.h. 3708 3709spdk_reactor_get_tsc_stats was added to return interesting statistics for each 3710reactor. 3711 3712### IOAT 3713 3714IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable 3715option with "Yes" in `[Ioat]` section of the configuration file. The Disable option is 3716now deprecated and will be removed in a future release. 3717 3718## v18.04: Logical Volume Snapshot/Clone, iSCSI Initiator, Bdev QoS, VPP Userspace TCP/IP 3719 3720### vhost 3721 3722The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the 3723DPDK rte_vhost vulnerability [CVE-2018-1059](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1059). 3724Please see this [security advisory](https://access.redhat.com/security/cve/cve-2018-1059) 3725for additional information on the DPDK vulnerability. 3726 3727Workarounds have been added to ensure vhost compatibility with QEMU 2.12. 3728 3729EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the 3730[vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3731 3732### Unified Target Application 3733 3734A new unified SPDK target application, `spdk_tgt`, has been added. This application combines the 3735functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target, 3736and vhost target. The new application can be managed through the existing configuration file and 3737[JSON-RPC](http://www.spdk.io/doc/jsonrpc.html) methods. 3738 3739### Env 3740 3741spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk(). 3742 3743New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added. 3744These new functions have a `flags` parameter that allows the user to specify whether the allocated 3745memory needs to be suitable for DMA and whether it should be shared across processes with the same 3746shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will 3747eventually be deprecated and removed. 3748 3749### Bdev 3750 3751A new optional bdev module interface function, `init_complete`, has been added to notify bdev modules 3752when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require 3753notification that the set of initialization examine() calls is complete. 3754 3755The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with 3756the spdk_bdev_get_uuid() function. 3757 3758Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the 3759[set_bdev_qos_limit_iops](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit_iops) documentation 3760for more details. 3761 3762### RPC 3763 3764The `[Rpc]` configuration file section, which was deprecated in v18.01, has been removed. 3765Users should switch to the `-r` command-line parameter instead. 3766 3767The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as 3768needed; previously, the response was limited to 32 kilobytes. 3769 3770### SPDKCLI 3771 3772EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available. 3773See the [SPDKCLI](http://www.spdk.io/doc/spdkcli.html) documentation for more details. 3774 3775### NVMe Driver 3776 3777EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is 3778experimental pending a functional allocator to free and reallocate CMB buffers. 3779 3780spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available. 3781 3782New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns() 3783have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if 3784a namespace ID is active. 3785 3786### NVMe-oF Target 3787 3788Namespaces may now be assigned unique identifiers via new optional `eui64` and `nguid` parameters 3789to the `nvmf_subsystem_add_ns` RPC method. Additionally, the NVMe-oF target automatically exposes 3790the backing bdev's UUID as the namespace UUID when available. 3791 3792spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion. 3793 3794### Blobstore 3795 3796A number of functions have been renamed: 3797 3798- spdk_bs_io_write_blob() => spdk_blob_io_write() 3799- spdk_bs_io_read_blob() => spdk_blob_io_read() 3800- spdk_bs_io_writev_blob() => spdk_blob_io_writev() 3801- spdk_bs_io_readv_blob() => spdk_blob_io_readv() 3802- spdk_bs_io_unmap_blob() => spdk_blob_io_unmap() 3803- spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes() 3804 3805The old names still exist but are deprecated. They will be removed in the v18.07 release. 3806 3807spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O 3808are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still 3809required to sync the updated metadata to disk. 3810 3811### Logical Volumes 3812 3813A new `destroy_lvol_bdev` RPC method to delete logical volumes has been added. 3814 3815Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination. 3816 3817New Snapshot and Clone functionalities have been added. User may create Snapshots of existing Lvols 3818and Clones of existing Snapshots. 3819See the [lvol snapshots](http://www.spdk.io/doc/logical_volumes.html#lvol_snapshots) documentation 3820for more details. 3821 3822Resizing logical volumes is now supported via the `resize_lvol_bdev` RPC method. 3823 3824### Lib 3825 3826A set of changes were made in the SPDK's lib code altering 3827instances of calls to `exit()` and `abort()` to return a failure instead 3828wherever reasonably possible. 3829 3830spdk_app_start() no longer exit()'s on an internal failure, but 3831instead returns a non-zero error status. 3832 3833spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid 3834option, but instead returns SPDK_APP_PARSE_ARGS_HELP and 3835SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS 3836on success. 3837 3838spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07. 3839 3840### I/O Channels 3841 3842The prototype for spdk_poller_fn() has been modified; it now returns a value indicating 3843whether or not the poller did any work. Existing pollers will need to be updated to 3844return a value. 3845 3846### iSCSI Target 3847 3848The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace 3849TCP/IP stack. See the [iSCSI VPP documentation](http://www.spdk.io/doc/iscsi.html#vpp) for more 3850details. 3851 3852### iSCSI initiator 3853 3854An iSCSI initiator bdev module has been added to SPDK. This module should be considered 3855experimental pending additional features and tests. More details can be found in 3856lib/bdev/iscsi/README. 3857 3858### PMDK 3859 3860The persistent memory (PMDK) bdev module is now enabled using `--with-pmdk` instead of 3861`--with-nvml`. This reflects the renaming of the persistent memory library from NVML to 3862PMDK. 3863 3864### Virtio Block driver 3865 3866A userspace driver for Virtio Block devices has been added. It was built on top of the 3867[Virtio](http://www.spdk.io/doc/virtio.html) library and can be managed similarly to 3868the Virtio SCSI driver. See the 3869[Virtio Block](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_blk) reference for 3870more information. 3871 3872### Virtio with 2MB hugepages 3873 3874The previous 1GB hugepage limitation has now been lifted. A new `-g` command-line option 3875enables SPDK Virtio to work with 2MB hugepages. 3876See [2MB hugepages](http://www.spdk.io/doc/virtio.html#virtio_2mb) for details. 3877 3878## v18.01: Blobstore Thin Provisioning 3879 3880### Build System 3881 3882The build system now includes a `make install` rule, including support for the common 3883`DESTDIR` and `prefix` variables as used in other build systems. Additionally, the prefix 3884may be set via the configure `--prefix` option. Example: `make install prefix=/usr`. 3885 3886### RPC 3887 3888A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/tmp/spdk.sock. 3889A -r option command line option has been added to enable an alternative UNIX domain socket location, 3890or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file 3891section is now deprecated and will be removed in the v18.04 release. 3892 3893### I/O Channels 3894 3895spdk_poller_register() and spdk_poller_unregister() were moved from the event 3896framework (include/spdk/event.h) to the I/O channel library 3897(include/spdk/io_channel.h). This allows code that doesn't depend on the event 3898framework to request registration and unregistration of pollers. 3899 3900spdk_for_each_channel() now allows asynchronous operations during iteration. 3901Instead of immediately continuing the interaction upon returning from the iteration 3902callback, the user must call spdk_for_each_channel_continue() to resume iteration. 3903 3904### Block Device Abstraction Layer (bdev) 3905 3906The poller abstraction was removed from the bdev layer. There is now a general purpose 3907abstraction for pollers available in include/spdk/io_channel.h 3908 3909### Lib 3910 3911A set of changes were made in the SPDK's lib code altering, 3912instances of calls to `exit()` and `abort()` to return a failure instead 3913wherever reasonably possible. This has resulted in return type changes of 3914the API for: 3915 3916- spdk_env_init() from type `void` to `int`. 3917- spdk_mem_map_init() from type `void` to `int`. 3918 3919Applications making use of these APIs should be modified to check for 3920a non-zero return value instead of relying on them to fail without return. 3921 3922### NVMe Driver 3923 3924SPDK now supports hotplug for vfio-attached devices. But there is one thing keep in mind: 3925Only physical removal events are supported; removing devices via the sysfs `remove` file will not work. 3926 3927### NVMe-oF Target 3928 3929Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available 3930cores round-robin. The "Core" option in the configuration file has been removed. 3931 3932### Blobstore 3933 3934A number of functions have been renamed: 3935 3936- spdk_bs_md_resize_blob() => spdk_blob_resize() 3937- spdk_bs_md_sync_blob() => spdk_blob_sync_md() 3938- spdk_bs_md_close_blob() => spdk_blob_close() 3939- spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names() 3940- spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value() 3941- spdk_blob_md_set_xattr() => spdk_blob_set_xattr() 3942- spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr() 3943- spdk_bs_md_create_blob() => spdk_bs_create_blob() 3944- spdk_bs_md_open_blob() => spdk_bs_open_blob() 3945- spdk_bs_md_delete_blob() => spdk_bs_delete_blob() 3946- spdk_bs_md_iter_first() => spdk_bs_iter_first() 3947- spdk_bs_md_iter_next() => spdk_bs_iter_next() 3948 3949The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument 3950rather than struct spdk_blob **. 3951 3952The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument 3953rather than struct spdk_blob **. 3954 3955Thin provisioning support has been added to the blobstore. It can be enabled by setting the 3956`thin_provision` flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext(). 3957 3958### NBD device 3959 3960The NBD application (test/lib/bdev/nbd) has been removed; Same functionality can now be 3961achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method. 3962See the [GPT](http://www.spdk.io/doc/bdev.html#bdev_config_gpt) documentation for more details. 3963 3964### FIO plugin 3965 3966SPDK `fio_plugin` now supports FIO 3.3. The support for previous FIO 2.21 has been dropped, 3967although it still remains to work for now. The new FIO contains huge amount of bugfixes and 3968it's recommended to do an update. 3969 3970### Virtio library 3971 3972Previously a part of the bdev_virtio module, now a separate library. Virtio is now available 3973via `spdk_internal/virtio.h` file. This is an internal interface to be used when implementing 3974new Virtio backends, namely Virtio-BLK. 3975 3976### iSCSI 3977 3978The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated 3979to an epoll/kqueue descriptor on the master core when idle. 3980 3981## v17.10: Logical Volumes 3982 3983### New dependencies 3984 3985libuuid was added as new dependency for logical volumes. 3986 3987libnuma is now required unconditionally now that the DPDK submodule has been updated to DPDK 17.08. 3988 3989### Block Device Abstraction Layer (bdev) 3990 3991An [fio](http://github.com/axboe/fio) plugin was added that can route 3992I/O to the bdev layer. See the [plugin documentation](https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin/) 3993for more information. 3994 3995spdk_bdev_unmap() was modified to take an offset and a length in bytes as 3996arguments instead of requiring the user to provide an array of SCSI 3997unmap descriptors. This limits unmaps to a single contiguous range. 3998 3999spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out. 4000If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using 4001write commands. 4002 4003New API functions that accept I/O parameters in units of blocks instead of bytes 4004have been added: 4005 4006- spdk_bdev_read_blocks(), spdk_bdev_readv_blocks() 4007- spdk_bdev_write_blocks(), spdk_bdev_writev_blocks() 4008- spdk_bdev_write_zeroes_blocks() 4009- spdk_bdev_unmap_blocks() 4010 4011The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be 4012retried later. 4013 4014### Linux AIO bdev 4015 4016The AIO bdev now allows the user to override the auto-detected block size. 4017 4018### NVMe driver 4019 4020The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field. 4021NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take 4022into account when splitting I/O requests. 4023 4024The HotplugEnable option in `[Nvme]` sections of the configuration file is now 4025"No" by default. It was previously "Yes". 4026 4027The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the 4028NVMe Controller associated with a given namespace. 4029 4030The NVMe library now allows the user to specify a host identifier when attaching 4031to a controller. The host identifier is used as part of the Reservations feature, 4032as well as in the NVMe-oF Connect command. The default host ID is also now a 4033randomly-generated UUID, and the default host NQN uses the host ID to generate 4034a UUID-based NQN. 4035 4036spdk_nvme_connect() was added to allow the user to connect directly to a single 4037NVMe or NVMe-oF controller. 4038 4039### NVMe-oF Target (nvmf_tgt) 4040 4041The NVMe-oF target no longer requires any in-capsule data buffers to run, and 4042the feature is now entirely optional. Previously, at least 4 KiB in-capsule 4043data buffers were required. 4044 4045NVMe-oF subsystems have a new configuration option, AllowAnyHost, to control 4046whether the host NQN whitelist is enforced when accepting new connections. 4047If no Host options have been specified and AllowAnyHost is disabled, the 4048connection will be denied; this is a behavior change from previous releases, 4049which allowed any host NQN to connect if the Host list was empty. 4050AllowAnyHost is disabled by default. 4051 4052NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number 4053of namespaces per subsystem is no longer limited. 4054 4055The NVMe-oF target now supports the Write Zeroes command. 4056 4057### Environment Abstraction Layer 4058 4059A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide 4060additional clarity when constructing spdk_mempools. Previously, -1 could be 4061passed and the library would choose a reasonable default, but this new value 4062makes it explicit that the default is being used. 4063 4064### Blobstore 4065 4066The blobstore super block now contains a bstype field to identify the type of the blobstore. 4067Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load(). 4068 4069spdk_bs_destroy() was added to allow destroying blobstore on device 4070with an initialized blobstore. 4071 4072spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable 4073scattered payloads. 4074 4075A CLI tool for blobstore has been added, allowing basic operations through either command 4076line or shell interface. See the [blobcli](https://github.com/spdk/spdk/tree/master/examples/blob/cli) 4077documentation for more details. 4078 4079### Event Framework 4080 4081The ability to set a thread name, previously only used by the reactor code, is 4082now part of the spdk_thread_allocate() API. Users may specify a thread name 4083which will show up in tools like `gdb`. 4084 4085### Log 4086 4087The spdk_trace_dump() function now takes a new parameter to allow the caller to 4088specify an output file handle (stdout or stderr, for example). 4089 4090### Logical Volumes 4091 4092Logical volumes library built on top of SPDK blobstore has been added. 4093It is possible to create logical volumes on top of other devices using RPC. 4094 4095See the [logical volumes](http://www.spdk.io/doc/logical_volumes.html) documentation for more information. 4096 4097### Persistent Memory 4098 4099A new persistent memory bdev type has been added. 4100The persistent memory block device is built on top of [libpmemblk](http://pmem.io/nvml/libpmemblk/). 4101It is possible to create pmem devices on top of pmem pool files using RPC. 4102 4103See the [Pmem Block Device](http://www.spdk.io/doc/bdev.html#bdev_config_pmem) documentation for more information. 4104 4105### Virtio SCSI driver 4106 4107A userspace driver for Virtio SCSI devices has been added. 4108The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application. 4109 4110See the [Virtio SCSI](http://www.spdk.io/doc/virtio.html) documentation and 4111[Getting Started](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_scsi) guide for more information. 4112 4113### Vhost target 4114 4115The vhost target application now supports live migration between QEMU instances. 4116 4117## v17.07: Build system improvements, userspace vhost-blk target, and GPT bdev 4118 4119### Build System 4120 4121A `configure` script has been added to simplify the build configuration process. 4122The existing CONFIG file and `make CONFIG_...` options are also still supported. 4123Run `./configure --help` for information about available configuration options. 4124 4125A DPDK submodule has been added to make building SPDK easier. If no `--with-dpdk` 4126option is specified to configure, the SPDK build system will automatically build a 4127known-good configuration of DPDK with the minimal options enabled. See the Building 4128section of README.md for more information. 4129 4130A [Vagrant](https://www.vagrantup.com/) setup has been added to make it easier to 4131develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant 4132section of README.md for more information. 4133 4134### Userspace vhost-blk target 4135 4136The vhost library and example app have been updated to support the vhost-blk 4137protocol in addition to the existing vhost-scsi protocol. 4138See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 4139 4140### Block device abstraction layer (bdev) 4141 4142A GPT virtual block device has been added, which automatically exposes GPT partitions 4143with a special SPDK-specific partition type as bdevs. 4144See the [GPT bdev documentation](http://www.spdk.io/doc/bdev.md#bdev_config_gpt) for 4145more information. 4146 4147### NVMe driver 4148 4149The NVMe driver has been updated to support recent Intel SSDs, including the Intel® 4150Optane™ SSD DC P4800X series. 4151 4152A workaround has been added for devices that failed to recognize register writes 4153during controller reset. 4154 4155The NVMe driver now allocates request tracking objects on a per-queue basis. The 4156number of requests allowed on an I/O queue may be set during `spdk_nvme_probe()` by 4157modifying `io_queue_requests` in the opts structure. 4158 4159The SPDK NVMe `fio_plugin` has been updated to support multiple threads (`numjobs`). 4160 4161spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override 4162controller-level options for each individual I/O queue pair. 4163Existing callers with qprio == 0 can be updated to: 4164~~~ 4165... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0); 4166~~~ 4167Callers that need to specify a non-default qprio should be updated to: 4168~~~ 4169struct spdk_nvme_io_qpair_opts opts; 4170spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts)); 4171opts.qprio = SPDK_NVME_QPRIO_...; 4172... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts)); 4173~~~ 4174 4175### Environment Abstraction Layer 4176 4177The environment abstraction layer has been updated to include several new functions 4178in order to wrap additional DPDK functionality. See `include/spdk/env.h` for the 4179current set of functions. 4180 4181### SPDK Performance Analysis with Intel® VTune™ Amplifier 4182 4183Support for SPDK performance analysis has been added to Intel® VTune™ Amplifier 2018. 4184 4185This analysis provides: 4186 4187- I/O performance monitoring (calculating standard I/O metrics like IOPS, throughput, etc.) 4188- Tuning insights on the interplay of I/O and compute devices by estimating how many cores 4189 would be reasonable to provide for SPDK to keep up with a current storage workload. 4190 4191See the VTune Amplifier documentation for more information. 4192 4193## v17.03: Blobstore and userspace vhost-scsi target 4194 4195### Blobstore and BlobFS 4196 4197The blobstore is a persistent, power-fail safe block allocator designed to be 4198used as the local storage system backing a higher-level storage service. 4199See the [blobstore documentation](http://www.spdk.io/doc/blob.html) for more details. 4200 4201BlobFS adds basic filesystem functionality like filenames on top of the blobstore. 4202This release also includes a RocksDB Env implementation using BlobFS in place of the 4203kernel filesystem. 4204See the [BlobFS documentation](http://www.spdk.io/doc/blobfs.html) for more details. 4205 4206### Userspace vhost-scsi target 4207 4208A userspace implementation of the QEMU vhost-scsi protocol has been added. 4209The vhost target is capable of exporting SPDK bdevs to QEMU-based VMs as virtio devices. 4210See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 4211 4212### Event framework 4213 4214The overhead of the main reactor event loop was reduced by optimizing the number of 4215calls to spdk_get_ticks() per iteration. 4216 4217### NVMe library 4218 4219The NVMe library will now automatically split readv/writev requests with scatter-gather 4220lists that do not map to valid PRP lists when the NVMe controller does not natively 4221support SGLs. 4222 4223The `identify` and `perf` NVMe examples were modified to add a consistent format for 4224specifying remote NVMe over Fabrics devices via the `-r` option. 4225This is implemented using the new `spdk_nvme_transport_id_parse()` function. 4226 4227### iSCSI Target 4228 4229The [Nvme] section of the configuration file was modified to remove the `BDF` directive 4230and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 4231devices can now be specified as the backing block device. A script to generate an 4232entire [Nvme] section based on the local NVMe devices attached was added at 4233`scripts/gen_nvme.sh`. 4234 4235### NVMe-oF Target 4236 4237The [Nvme] section of the configuration file was modified to remove the `BDF` directive 4238and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 4239devices can now be specified as the backing block device. A script to generate an 4240entire [Nvme] section based on the local NVMe devices attached was added at 4241`scripts/gen_nvme.sh`. 4242 4243## v16.12: NVMe over Fabrics host, hotplug, and multi-process 4244 4245### NVMe library 4246 4247The NVMe library has been changed to create its own request memory pool rather than 4248requiring the user to initialize the global `request_mempool` variable. Apps can be 4249updated by simply removing the initialization of `request_mempool`. Since the NVMe 4250library user no longer needs to know the size of the internal NVMe request 4251structure to create the pool, the `spdk_nvme_request_size()` function was also removed. 4252 4253The `spdk_nvme_ns_cmd_deallocate()` function was renamed and extended to become 4254`spdk_nvme_ns_cmd_dataset_management()`, which allows access to all of the NVMe 4255Dataset Management command's parameters. Existing callers can be updated to use 4256`spdk_nvme_ns_cmd_dataset_management()` with `SPDK_NVME_DSM_ATTR_DEALLOCATE` as the 4257`type` parameter. 4258 4259The NVMe library SGL callback prototype has been changed to return virtual addresses 4260rather than physical addresses. Callers of `spdk_nvme_ns_cmd_readv()` and 4261`spdk_nvme_ns_cmd_writev()` must update their `next_sge_fn` callbacks to match. 4262 4263The NVMe library now supports NVMe over Fabrics devices in addition to the existing 4264support for local PCIe-attached NVMe devices. For an example of how to enable 4265NVMe over Fabrics support in an application, see `examples/nvme/identify` and 4266`examples/nvme/perf`. 4267 4268Hot insert/remove support for NVMe devices has been added. To enable NVMe hotplug 4269support, an application should call the `spdk_nvme_probe()` function on a regular 4270basis to probe for new devices (reported via the existing `probe_cb` callback) and 4271removed devices (reported via a new `remove_cb` callback). Hotplug is currently 4272only supported on Linux with the `uio_pci_generic` driver, and newly-added NVMe 4273devices must be bound to `uio_pci_generic` by an external script or tool. 4274 4275Multiple processes may now coordinate and use a single NVMe device simultaneously 4276using [DPDK Multi-process Support](http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html). 4277 4278### NVMe over Fabrics target (`nvmf_tgt`) 4279 4280The `nvmf_tgt` configuration file format has been updated significantly to enable 4281new features. See the example configuration file `etc/spdk/nvmf.conf.in` for 4282more details on the new and changed options. 4283 4284The NVMe over Fabrics target now supports virtual mode subsystems, which allow the 4285user to export devices from the SPDK block device abstraction layer as NVMe over 4286Fabrics subsystems. Direct mode (raw NVMe device access) is also still supported, 4287and a single `nvmf_tgt` may export both types of subsystems simultaneously. 4288 4289### Block device abstraction layer (bdev) 4290 4291The bdev layer now supports scatter/gather read and write I/O APIs, and the NVMe 4292blockdev driver has been updated to support scatter/gather. Apps can use the 4293new scatter/gather support via the `spdk_bdev_readv()` and `spdk_bdev_writev()` 4294functions. 4295 4296The bdev status returned from each I/O has been extended to pass through NVMe 4297or SCSI status codes directly in cases where the underlying device can provide 4298a more specific status code. 4299 4300A Ceph RBD (RADOS Block Device) blockdev driver has been added. This allows the 4301`iscsi_tgt` and `nvmf_tgt` apps to export Ceph RBD volumes as iSCSI LUNs or 4302NVMe namespaces. 4303 4304### General changes 4305 4306`libpciaccess` has been removed as a dependency and DPDK PCI enumeration is 4307used instead. Prior to DPDK 16.07 enumeration by class code was not supported, 4308so for earlier DPDK versions, only Intel SSD DC P3x00 devices will be discovered 4309by the NVMe library. 4310 4311The `env` environment abstraction library has been introduced, and a default 4312DPDK-based implementation is provided as part of SPDK. The goal of the `env` 4313layer is to enable use of alternate user-mode memory allocation and PCI access 4314libraries. See `doc/porting.md` for more details. 4315 4316The build process has been modified to produce all of the library files in the 4317`build/lib` directory. This is intended to simplify the use of SPDK from external 4318projects, which can now link to SPDK libraries by adding the `build/lib` directory 4319to the library path via `-L` and linking the SPDK libraries by name (for example, 4320`-lspdk_nvme -lspdk_log -lspdk_util`). 4321 4322`nvmf_tgt` and `iscsi_tgt` now have a JSON-RPC interface, which allows the user 4323to query and modify the configuration at runtime. The RPC service is disabled by 4324default, since it currently does not provide any authentication or security 4325mechanisms; it should only be enabled on systems with controlled user access 4326behind a firewall. An example RPC client implemented in Python is provided in 4327`scripts/rpc.py`. 4328 4329## v16.08: iSCSI target, NVMe over Fabrics maturity 4330 4331This release adds a userspace iSCSI target. The iSCSI target is capable of exporting 4332NVMe devices over a network using the iSCSI protocol. The application is located 4333in app/iscsi_tgt and a documented configuration file can be found at etc/spdk/spdk.conf.in. 4334 4335This release also significantly improves the existing NVMe over Fabrics target. 4336 4337- The configuration file format was changed, which will require updates to 4338 any existing nvmf.conf files (see `etc/spdk/nvmf.conf.in`): 4339 - `SubsystemGroup` was renamed to `Subsystem`. 4340 - `AuthFile` was removed (it was unimplemented). 4341 - `nvmf_tgt` was updated to correctly recognize NQN (NVMe Qualified Names) 4342 when naming subsystems. The default node name was changed to reflect this; 4343 it is now "nqn.2016-06.io.spdk". 4344 - `Port` and `Host` sections were merged into the `Subsystem` section 4345 - Global options to control max queue depth, number of queues, max I/O 4346 size, and max in-capsule data size were added. 4347 - The Nvme section was removed. Now a list of devices is specified by 4348 bus/device/function directly in the Subsystem section. 4349 - Subsystems now have a Mode, which can be Direct or Virtual. This is an attempt 4350 to future-proof the interface, so the only mode supported by this release 4351 is "Direct". 4352- Many bug fixes and cleanups were applied to the `nvmf_tgt` app and library. 4353- The target now supports discovery. 4354 4355This release also adds one new feature and provides some better examples and tools 4356for the NVMe driver. 4357 4358- The Weighted Round Robin arbitration method is now supported. This allows 4359 the user to specify different priorities on a per-I/O-queue basis. To 4360 enable WRR, set the `arb_mechanism` field during `spdk_nvme_probe()`. 4361- A simplified "Hello World" example was added to show the proper way to use 4362 the NVMe library API; see `examples/nvme/hello_world/hello_world.c`. 4363- A test for measuring software overhead was added. See `test/lib/nvme/overhead`. 4364 4365## v16.06: NVMf userspace target 4366 4367This release adds a userspace NVMf (NVMe over Fabrics) target, conforming to the 4368newly-released NVMf 1.0/NVMe 1.2.1 specification. The NVMf target exports NVMe 4369devices from a host machine over the network via RDMA. Currently, the target is 4370limited to directly exporting physical NVMe devices, and the discovery subsystem 4371is not supported. 4372 4373This release includes a general API cleanup, including renaming all declarations 4374in public headers to include a `spdk` prefix to prevent namespace clashes with 4375user code. 4376 4377- NVMe 4378 - The `nvme_attach()` API was reworked into a new probe/attach model, which 4379 moves device detection into the NVMe library. The new API also allows 4380 parallel initialization of NVMe controllers, providing a major reduction in 4381 startup time when using multiple controllers. 4382 - I/O queue allocation was changed to be explicit in the API. Each function 4383 that generates I/O requests now takes a queue pair (`spdk_nvme_qpair *`) 4384 argument, and I/O queues may be allocated using 4385 `spdk_nvme_ctrlr_alloc_io_qpair()`. This allows more flexible assignment of 4386 queue pairs than the previous model, which only allowed a single queue 4387 per thread and limited the total number of I/O queues to the lowest number 4388 supported on any attached controller. 4389 - Added support for the Write Zeroes command. 4390 - `examples/nvme/perf` can now report I/O command latency from the 4391 the controller's viewpoint using the Intel vendor-specific read/write latency 4392 log page. 4393 - Added namespace reservation command support, which can be used to coordinate 4394 sharing of a namespace between multiple hosts. 4395 - Added hardware SGL support, which enables use of scattered buffers that 4396 don't conform to the PRP list alignment and length requirements on supported 4397 NVMe controllers. 4398 - Added end-to-end data protection support, including the ability to write and 4399 read metadata in extended LBA (metadata appended to each block of data in the 4400 buffer) and separate metadata buffer modes. 4401 See `spdk_nvme_ns_cmd_write_with_md()` and `spdk_nvme_ns_cmd_read_with_md()` 4402 for details. 4403- IOAT 4404 - The DMA block fill feature is now exposed via the `ioat_submit_fill()` 4405 function. This is functionally similar to `memset()`, except the memory is 4406 filled with an 8-byte repeating pattern instead of a single byte like memset. 4407- PCI 4408 - Added support for using DPDK for PCI device mapping in addition to the 4409 existing libpciaccess option. Using the DPDK PCI support also allows use of 4410 the Linux VFIO driver model, which means that SPDK userspace drivers will work 4411 with the IOMMU enabled. Additionally, SPDK applications may be run as an 4412 unprivileged user with access restricted to a specific set of PCIe devices. 4413 - The PCI library API was made more generic to abstract away differences 4414 between the underlying PCI access implementations. 4415 4416## v1.2.0: IOAT user-space driver 4417 4418This release adds a user-space driver with support for the Intel I/O Acceleration Technology 4419(I/OAT, also known as "Crystal Beach") DMA offload engine. 4420 4421- IOAT 4422 - New user-space driver supporting DMA memory copy offload 4423 - Example programs `ioat/perf` and `ioat/verify` 4424 - Kernel-mode DMA engine test driver `kperf` for performance comparison 4425- NVMe 4426 - Per-I/O flags for Force Unit Access (FUA) and Limited Retry 4427 - Public API for retrieving log pages 4428 - Reservation register/acquire/release/report command support 4429 - Scattered payload support - an alternate API to provide I/O buffers via a sequence of callbacks 4430 - Declarations and `nvme/identify` support for Intel SSD DC P3700 series vendor-specific log pages and features 4431- Updated to support DPDK 2.2.0 4432 4433## v1.0.0: NVMe user-space driver 4434 4435This is the initial open source release of the Storage Performance Development Kit (SPDK). 4436 4437Features: 4438 4439- NVMe user-space driver 4440- NVMe example programs 4441 - `examples/nvme/perf` tests performance (IOPS) using the NVMe user-space driver 4442 - `examples/nvme/identify` displays NVMe controller information in a human-readable format 4443- Linux and FreeBSD support 4444