1# Changelog 2 3## v24.05: (Upcoming Release) 4 5### accel 6 7spdk_accel_get/set_opts() has changed to act more like spdk_bdev's variant. 8 9### bdev 10 11Added `spdk_bdev_get_nvme_ctratt()` API to get controller attributes of bdev. 12 13### lvol 14 15Added `bdev_lvol_start_shallow_copy` RPC to start a shallow copy of an lvol over a given bdev and 16`bdev_lvol_check_shallow_copy` RPC to get the status of the operation. 17These RPCs use the new API `spdk_lvol_shallow_copy`. 18 19### blobstore 20 21Added new API `spdk_bs_blob_shallow_copy` to make a shallow copy from a blob to a blobstore device. Only clusters 22allocated to the blob will be written on the device. 23 24### event 25 26SPDK applications can now start with `--wait-for-rpc` and JSON configuration provided at the same time. 27 28Struct `spdk_app_opts` was extended by `json_data` and `json_data_size` fields 29allowing user to provide a buffer with JSON config as alternative to providing JSON config file path 30(these fields are mutually exclusive). 31 32### init 33 34Function `spdk_subsystem_init_from_json_config` is deprecated and will be removed in 24.09 release. 35Please use `spdk_subsystem_load_config` instead. 36 37### nvme 38 39Added `spdk_nvme_ctrlr_get_max_sges`. 40 41### bdev_uring 42 43Added `bdev_uring_rescan` RPC to allow rescaning the size of uring bdev. 44 45### iscsi 46 47Added `iscsi_get_stats` RPC method to get stat information about all active connections. 48 49Added `iscsi_enable_histogram` RPC method to enable or disable histogram for specified iscsi target. 50 51Added `iscsi_get_histogram` RPC method to get histogram for specified iscsi target. 52 53### thread 54 55spdk_iobuf_get/set_opts() has changed to act more like spdk_bdev's variant. 56 57### trace 58 59Merged `struct spdk_trace_flags` and `struct spdk_trace_histories` into 60new `struct spdk_trace_file`. Also renamed `spdk_get_trace_histories_size()` 61to `spdk_get_trace_file_size()`. 62 63### nvmf 64 65Added support for namespace masking using new C APIs `spdk_nvmf_ns_add_host` and 66`spdk_nvmf_ns_remove_host` and RPCs `nvmf_ns_add_host` and `nvmf_ns_remove_host`. 67Users must pass new `--no-auto-visible` parameter to `nvmf_subsystem_add_ns` RPC to allow 68namespace masking to be controlled by these new RPCs. 69 70Added `spdk_nvmf_subsystem_get_ana_state` API to get current ANA state of 71a particular ANA group ID. 72 73### spdk_trace 74 75`spdk_trace` has learned how to use the most recent trace file in /dev/shm when 76the user has specified neither `-f` nor `-s` options. This is only available on 77Linux since FreeBSD does not mount have a /dev/shm mount. 78 79## v24.01 80 81### accel 82 83Added support for new operations for computing Data Integrity Field (DIF), 84`spdk_accel_submit_dif_verify()`, `spdk_accel_submit_dif_generate()` and 85`spdk_accel_submit_dif_generate_copy()`. 86 87Added `spdk_accel_get_opcode_name()` API that returns name of an operation. 88 89Added `spdk_accel_get_module()` API that returns pointer to a module. 90 91Added optional `init()` and `fini()` callbacks to `spdk_accel_driver` in order to allow 92drivers to initialize and clean up resources by the driver. 93 94### accel_error 95 96Added new `error` accel module that allows user to inject error to selected accel operations. 97 98### bdev 99 100Added `spdk_bdev_nvme_iov_passthru_md()` API to allow sending larger passthru commands. 101 102Added `max_rw_size` field to `spdk_bdev` structure allowing bdev modules to specify maximum 103size of an I/O. 104 105Added `iobuf_small_cache_size` and `iobuf_large_cache_size` parameters to 106`bdev_set_options` RPC making iobuf use in bdev configurable. 107 108### bdev_raid 109 110Added support for rebuild in raid1 and raid5f levels. 111 112Added `superblock` argument to `bdev_raid_create` RPC to allow writing out raid metadata to 113a superblock. It can be then re-created from examined base bdevs, without the need 114to issue `bdev_raid_create`. 115 116Added `bdev_raid_add_base_bdev` RPC to add a base bdevs to an existing raid bdev. 117 118Added `bdev_raid_set_options` RPC that changes options for raid bdev module, 119currently allows setting `process_window_size_kb` affecting background operations 120like rebuild. 121 122### blobstore 123 124Added `spdk_bs_grow_live()` and `spdk_bdev_update_bs_blockcnt()` API that can be used to 125increase size of blobstore filling the underlying device without first closing the blobstore. 126 127### env 128 129Added SPDK command line parameter `--no-huge`, which enables SPDK to run without hugepages. 130 131### event 132 133Added SPDK command line parameter `--interrupt-mode`, which enables to run SPDK in interrupt 134mode. Effective only if all components used in the application, support interrupt mode. 135 136### ftl 137 138Added `rpc_bdev_ftl_get_properties` and `bdev_ftl_set_property` RPC with matching API calls, 139to allow modification of FTL device properties. 140 141### idxd 142 143Added `spdk_idxd_submit_dif_check()` and `spdk_idxd_submit_dif_insert()` API to support 144new operations for computing Data Integrity Field (DIF). 145 146### json 147 148Added `spdk_json_write_uuid()`, `spdk_json_write_named_uuid()` and `spdk_json_decode_uuid()` 149to help with writing and decoding UUID. 150 151### nvme 152 153A new transport option `rdma_max_cq_size` was added to limit indefinite growth of CQ size. 154 155Added `spdk_nvme_ctrlr_cmd_iov_raw_with_md()` API to allow sending larger passthru commands. 156 157### nvmf 158 159Added `max_discard_size_kib` and `max_write_zeroes_size_kib` to `nvmf_create_subsystem` RPC to set the 160maximum discard size and maximum write zeroes size. 161 162Added new optional `--ana-state` (or shortly `-n`) parameter to `nvmf_subsystem_add_listener` RPC. 163 164Added public APIs `spdk_nvmf_subsystem_get_ana_reporting()` and `spdk_nvmf_subsystem_set_ana_state()`, 165replacing the internal functions `nvmf_subsystem_get_ana_reporting()` and `nvmf_subsystem_set_ana_state()` 166respectively. 167 168Added support for NVMe-oF referrals in NVMe-oF target with addition of 169`spdk_nvmf_tgt_add_referral()` and `spdk_nvmf_tgt_remove_referral()` API and `nvmf_discovery_add_referral`, 170`nvmf_discovery_remove_referral` and `nvmf_discovery_get_referrals` RPC. 171 172Added `spdk_nvmf_set_custom_ns_reservation_ops()` API allowing to register custom handlers for reservation 173operations. 174 175### scsi 176 177Added support for `SPDK_SBC_WRITE_SAME_10` and `SPDK_SBC_WRITE_SAME_16`. 178 179### thread 180 181Added `iobuf_get_stats` RPC and `spdk_iobuf_get_stats()` API to track iobuf use across components. 182 183### trace 184 185Added `spdk_trace_register_user_thread()` to initialize trace environment and 186`spdk_trace_unregister_user_thread()` to de-initialize trace environment 187for a user created thread. 188 189Modified `spdk_trace_init()` to take number of user created threads as a parameter. 190 191### ublk 192 193Added `ublk_recover_disk` RPC to support recovering after restarting ublk target. 194 195### vhost 196 197Added `caw_iov` field to struct `spdk_scsi_task` to support SBC-3 compare_and_write IO. 198 199Added parameter `delay` to `vhost_create_scsi_controller` RPC which allows user to defer starting 200the vhost-scsi controller until adding the scsi target is completed. 201 202Added `vhost_start_scsi_controller` RPC to start vhost-scsi controller, it could be used to support 203live recovery feature of vhost-scsi target. 204 205### rpc 206 207Functions `spdk_rpc_listen`, `spdk_rpc_accept` and `spdk_rpc_close` are deprecated and will be 208removed in 24.09 release. Please use `spdk_rpc_server_listen`, `spdk_rpc_server_accept` and 209`spdk_rpc_server_close` instead. 210 211## v23.09 212 213### accel 214 215Added `spdk_accel_get_buf_align()` API to get minimum buffer alignment to execute a given operation. 216 217Added `get_operation_info()` callback for accel_modules to return constraints for a given operation. 218 219### bdev 220 221Added `spdk_bdev_open_async()` API to allow asynchronous bdev open that waits for specified time 222until the block device appears. 223 224### bdev_nvme 225 226Added `bdev_nvme_enable_controller` and `bdev_nvme_disable_controller` RPC to enable and disable 227NVMe controller. For NVMe multipath `cntlid` is used to select only one NVMe-oF controller. 228 229Added `cntlid` field to `bdev_nvme_reset_controller` RPC to support resetting only one NVMe-oF 230controller in multipath configuration. 231 232### bdev_raid 233 234Added `bdev_raid_remove_base_bdev` RPC to remove a base bdev from existing raid bdev. 235 236### dpdk 237 238Updated DPDK submodule to DPDK 23.07. 239 240### env 241 242The `phys_addr` parameter in `spdk_*_malloc()` functions is now invalid. Passing non-NULL value 243will return NULL from the functions. The parameter was deprecated in SPDK 19.04. 244For retrieving physical addresses, `spdk_vtophys()` should be used instead. 245 246### go 247 248Added JSON-RPC 2.0 client written in Go to ease communication with SPDK. 249Please see the documentation at `go/rpc/` and example at `examples/go/`. 250 251### init 252 253Options for the JSON-RPC server initialization were added. The options are defined via the 254`spdk_rpc_opts` structure and is passed to the existing API `spdk_rpc_initialize()` as a new 255argument. The options include `log_file` and `log_level`. 256 257### intel-ipsec-mb 258 259Updated intel-ipsec-mb submodule to v1.4. 260 261### jsonrpc 262 263New APIs, `spdk_jsonrpc_set_log_level` and `spdk_jsonrpc_set_log_file`, were added to enable 264logging JSON RPC calls history. 265 266### log 267 268New APIs, `spdk_flog` and `spdk_vflog`, were added to write messages to the specified log file. 269 270### nvme 271 272The `spdk_nvme_accel_fn_table` interface has been extended with callbacks allowing users to chain 273multiple accel operations. Users can now send requests with an existing accel sequence by setting 274the `accel_sequence` field in `spdk_nvme_ns_cmd_ext_io_opts` if the controller supports it (i.e. 275sets the `SPDK_NVME_CTRLR_ACCEL_SEQUENCE_SUPPORTED` flag) and the user implements the necessary 276`spdk_nvme_accel_fn_table` callbacks. 277 278The NVMe bdev will now advertise support for accel sequences if the `allow_accel_sequence` flag is 279set in `bdev_nvme_set_options` and the underlying controller also supports them. 280 281The TCP transport will now calculate data digest using the accel sequence APIs if they're available. 282 283Added `spdk_nvme_qpair_is_connected()` API to check the connection status. 284 285### nvmf 286 287The `spdk_nvmf_request::data` field has been removed: instead, clients should set 288`->iov` and `->iovcnt` appropriately, as nvmf request APIs now expect any data 289buffers to be described there. `spdk_nvmf_request_get_data()` has been removed. 290 291`transport` field in `listen_addresses` of `nvmf_get_subsystems` RPC is deprecated. 292`trtype` field should be used instead. `transport` field will be removed in 24.01 release. 293 294Added `spdk_nvmf_subsystem_add_listener_ext()` API for accepting new connections on the provided 295address, now allowing to specify additional options for the listener. 296 297Deprecated `spdk_nvmf_subsytem_any_listener_allowed()` API (to be removed in 24.01) and replaced with 298spdk_nvmf_subsystem_any_listener_allowed()` API fixing the typo. 299 300Added `spdk_nvmf_subsystem_is_discovery()` API to check whether a given susbystem is discovery subsystem. 301 302### scripts 303 304`setup.sh` now supports interactive mode for device selection and hugepage reservation. 305 306Added `backport.sh` script that formalizes process of backporting commits from latest SPDK, 307to specific release branches. Please see [documentation](https://spdk.io/doc/backporting.html). 308 309### thread 310 311Added `spdk_thread_is_app_thread()` API to check if specified spdk_thread is the app thread. 312 313### util 314 315Extended DIF support to CRC-32 and CRC-64 format of the Protection Information. 316 317Added `spdk_crc64_nvme()` and `spdk_crc32c_nvme()`, CRC-64 and CRC-32C checksums compatible with 318NVMe Protection Information. 319 320## v23.05 321 322### accel 323 324Added API `spdk_accel_submit_xor` to perform XOR. 325 326Configuration of `mlx5_pci` driver of `accel_dpdk_cryptodev` module was changed. Now key and key2 327should have equal length and key size in hexlified form can be either 32 bytes for 128b XTS key or 32864 bytes for 256b XTS key. Support of the wrapped crypto credentials mode for this driver is 329dropped, only plain text mode is supported. 330 331The accel library will now collect statistics on the number of executed operations and processed 332bytes. These statistics can be retreived using the `accel_get_stats` RPC. 333 334Added support for appending a crc32 calculation to an accel sequence. 335 336### bdev 337 338A new API `spdk_bdev_module_claim_bdev_desc` was added. Unlike `spdk_bdev_module_claim_bdev`, this 339function requires a bdev descriptor to be passed and the claim is automatically released when the 340descriptor is closed. It allows bdev modules to claim bdevs as a single writer, multiple writers, or 341multiple readers. 342 343New APIs `spdk_bdev_quiesce`, `spdk_bdev_unquiesce`, `spdk_bdev_quiesce_range` and 344`spdk_bdev_unquiesce_range` were added. These allow I/O to be quiesced on an entire bdev or 345a specified LBA range. 346 347Removed `spdk_bdev_ext_io_opts` from `spdk_bdev_io` and replaced it with fields that were missing in 348`spdk_bdev_io`. 349 350It is now possible to submit an I/O with an associated accel sequence. All operations in such 351sequence will be executed prior to sending the I/O (writes) or after its completion (reads). 352 353### env 354 355New functions `spdk_env_get_main_core` and `spdk_env_get_cpuset` were added. 356 357It is now possible to run applications on cores with IDs greater than 128 by passing a core map 358using the `--lcores` parameter. 359 360### examples 361 362`examples/nvme/perf` application now accepts `--use-every-core` parameter that changes 363the existing worker and namespace association logic to access every namespace from each worker. 364This replicates behavior of bdevperf application when `-C` option is provided. 365 366### gpt 367 368GPT bdevs now use the GPT Unique Partition ID as the bdev's UUID. 369 370### idxd 371 372Removed the `SPDK_IDXD_FLAG_PERSISTENT` flag. 373 374### lvol 375 376New API `spdk_lvol_iter_immediate_clones` was added to iterate the clones of an lvol. 377 378New APIs `spdk_lvol_get_by_uuid` and `spdk_lvol_get_by_names` to get lvols by the lvol's UUID or 379lvstore and lvol names. 380 381New `bdev_lvol_get_lvols` RPC to list logical volumes. This provides information about logical 382volumes without providing information about the bdevs. It is useful for listing the lvols 383associated with specific lvol stores and for listing lvols that are degraded and have no 384associated bdev. 385 386Added support for external snapshots making it possible to create clones of any (including non-lvol) 387bdev. Users can create such clones using the `bdev_lvol_clone_bdev` RPC. 388 389### nvme 390 391New API `spdk_nvme_ns_get_format_index` was added to calculate the exact format index, that 392was used to format the namespace. 393 394Added two new APIs `spdk_nvme_ns_cmd_io_mgmt_recv` and `spdk_nvme_ns_cmd_io_mgmt_send` to 395receive and send the I/O management commands. 396 397### nvmf 398 399New `spdk_nvmf_request_copy_to/from_buf()` APIs have been added, which support 400iovecs, unlike the deprecated `spdk_nvmf_request_get_data()`. 401 402Two functions related to Asynchronous Event and error handling have been made public: 403 404- `spdk_nvmf_ctrlr_async_event_error_event`, 405- `spdk_nvmf_ctrlr_abort_aer`. 406 407New `spdk_nvmf_transport_create_async` was added, it accepts a callback and callback argument. 408`spdk_nvmf_transport_create` is marked deprecated. 409 410Parameters `cb_fn` and `ctx` of `spdk_nvmf_qpair_disconnect` API are deprecated. These parameters 411will be removed in 23.09 release. 412 413Added a `secure_channel` parameter to the `nvmf_subsystem_add_listener` RPC. When true, all 414connections established via this listener will immediately attempt to establish a secure channel, 415prior to any authentication. Only valid for the TCP transport. 416 417Added two optional transport module callbacks: `subsystem_add_host()` and `subsystem_remove_host()`. 418These functions will notify a transport about adding/removing hosts' access. 419 420TLS PSK identity is now generated from subsystem NQN and host NQN. PSK interchange format is now 421expected as input, when configuring TLS in SPDK. TLS feature is considered experimental. 422 423### part 424 425New API `spdk_bdev_part_construct_ext` is added and allows the bdev's UUID to be specified. 426 427### sock 428 429Added a zero copy receive interface allowing users to provide buffers to a socket group (via 430`spdk_sock_group_provide_buf()`), which will then get filled with the next portions of the stream on 431each socket. After that, pointer to the buffer holding the next part of the data can be obtained 432using `spdk_sock_recv_next()`. 433 434Added a callback `get_key()` to `spdk_sock_impl_opts` structure. 435 436New function `spdk_sock_get_impl_name()` was added to retrieve current socket implementation name. 437 438### thread 439 440Added two new APIs `spdk_thread_bind` and `spdk_thread_is_bound` to bind or unbind `spdk_thread` 441to its current CPU core, and check bound status. 442 443### ublk 444 445Enabled support for `UBLK_F_NEED_GET_DATA`. The buffers are no longer pre-allocated during 446initialization, but are allocated on-demand. Additionally, they're now allocated from the `iobuf` 447buffer pools. 448 449### util 450 451New APIs `spdk_uuid_is_null` and `spdk_uuid_set_null` were added to compare and 452set UUID to NULL value. 453 454Added new functions, `spdk_fd_group_{nest,unnest}`, providing efficient way of adding one 455`fd_group`'s file descriptors to another `fd_group`. 456 457## v23.01 458 459### accel 460 461New library isa-l-crypto has been added, it is used by accel library in crypto operations. 462 463New functions `spdk_accel_submit_encrypt` and `spdk_accel_submit_decrypt` were added. 464 465New accel module `dpdk_cryptodev` has been added. It uses DPDK crypto PMD and support encrypt and 466decrypt operations. New RPC `dpdk_cryptodev_scan_accel_module` has been added to enable this accel module. 467 468New accel module `mlx5` was added. It implements crypto operations, enabled when SPDK is configured with 469RDMA provider is mlx5_dv and crypto support. 470 471Introduced the concept of chaining multiple accel operations together and executing them all in a 472single call. The operation can be chained via one of the `spdk_accel_append_*` functions and then 473executed using `spdk_accel_sequence_finish`. Currently, copy, decompress, decrypt, encrypt, and 474fill operations support this mode. This feature is considered experimental. 475 476### bdev 477 478A new API `spkd_bdev_part_submit_request_ext` was added to specify a custom completion callback. 479 480New APIs `spdk_bdev_for_each_channel` and `spdk_bdev_for_each_channel_continue` and 481associated function pointers were added to iterate each channel of the required bdev. 482 483The RPC `bdev_get_iostat` now allows a user to query the per channel IO statistics for 484required bdev, and displays maximum and minimum latencies and I/O error counts. 485 486New `spdk_bdev_copy_blocks` and `spdk_bdev_get_max_copy` APIs to support copy commands. 487 488A new API `spdk_bdev_io_get_submit_tsc` was added to get the submit_tsc of the bdev I/O. 489 490Bdevs will no longer have UUIDs generated based on timestamp and are responsible for 491setting this field themselves. Generation of UUIDs for NVMe bdevs may be enabled by 492running `bdev_nvme_set_options` RPC with `--generate-uuids` option. These identifiers 493are based on serial number and namespace ID and will always be the same for a given 494device. 495 496A new API `spdk_bdev_channel_get_histogram` was added to get the histogram of a specified 497channel for a bdev. 498 499Converted internal use of `pthread_mutex_t` to `struct spdk_spinlock`. Consumers of bdev 500API functions must be on an SPDK thread or the program will abort. It is now enforced 501that no internal bdev locks can be held when a poller or message goes off CPU. 502 503A new RPC `bdev_reset_iostat` was added to reset I/O statistics of bdevs. Note that if one 504consumer reset I/O statistics, it affects all other consumers. 505 506Add function pointers, `dump_device_stat_json` and `reset_device_stat` to the bdev module 507function table to display and reset I/O statistics specific for the module specific bdev 508context. 509 510New APIs `spdk_bdev_reset_io_stat`, `spdk_bdev_add_io_stat` and `spdk_bdev_dump_io_stat_json` 511were added to process I/O statistics outside the generic bdev layer, especially in bdev modules. 512 513### bdev_crypto 514 515vbdev_crypto is updated to use accel framework instead of DPDK PMDs. 516 517### bdev_daos 518 519New RPC `bdev_daos_resize` was added to allow resizing the daos bdev. 520 521### bdev_malloc 522 523Both of interleaved and separated metadata are now supported by the malloc bdev module. 524 525Protection information is now supported by the malloc bdev module. 526 527### bdev_nvme 528 529Updated `bdev_nvme_set_options` RPC (and rpc.py) to support the new `transport_tos` parameter. 530 531For the active-active policy of the multipath mode, in addition to the default round-robin path 532selector, the minimum queue depth path selector was added. The minimum queue depth path selector 533selects an I/O path according to the number of outstanding requests of each nvme qpair. 534 535Added RPCs `bdev_nvme_start_mdns_discovery`, `bdev_nvme_get_mdns_discovery_info` and 536`bdev_nvme_stop_mdns_discovery` to perform Avahi based mDNS discovery service, 537as per NVMe TP 8009 - Automated Discovery of Ethernet Discovery Controllers. 538 539A new option `nvme_error_stat` was added to the `bdev_nvme_set_options` RPC to enable 540collecting NVMe error counts. 541 542Added I/O statistics per I/O path to the NVMe bdev module for NVMe bdev multipath. It can be 543enabled by a new option `io_path_stat` of RPC `bdev_nvme_set_options`. 544 545Added RPC `bdev_nvme_get_path_iostat` to get I/O statistics for IO paths of the NVMe bdev. 546 547Added `selector` parameter to `bdev_nvme_set_multipath_policy` RPC to set path selector for multipath. 548Option `round_robin` and `queue_depth` are available. 549 550Added `rr_min_io` option to RPC `bdev_nvme_set_multipath_policy`. It switches I/O to 551another path after rr_min_io I/Os are routed to current io path for the round-robin 552path selector. 553 554Added option `--io-path-stat` for RPC `bdev_nvme_set_option` to enable collecting io path stat. 555 556### bdev_raid 557 558Changed `bdev_raid_get_bdevs` RPC output format to include raid_bdev details. 559 560### bdevperf 561 562Promoted the application to example to match similar programs: fio_plugin and perf. 563It can now be found inside `examples/bdev/bdevperf`. 564 565### env 566 567Added `spdk_mempool_mem_iter` that allows to get start address and length of each memory 568chunk in order to create app-specific resources. 569 570### event 571 572Added core lock file mechanism to prevent the same CPU cores from being used by multiple 573SPDK application instances. 574 575Added `--disable-cpumask-locks` command line switch to disable CPU locks on SPDK startup. 576 577Added RPCs `framework_enable_cpumask_locks` and `framework_disable_cpumask_locks` to enable 578and disable CPU core locks in runtime. 579 580Added `--rpcs-allowed` command line option. Users can specify a comma-separated list of RPC 581names with this option to restrict allowed RPCs to only that list. 582 583Added `--vfio-vf-token` command line option to specify a VF token (UUID) 584shared between SR-IOV PF and VFs for vfio_pci driver. 585 586The `spdk_iov_xfer_*()` APIs were added for handling incremental copying between 587an iovec and a buffer. 588 589### json 590 591Added API `spdk_json_write_double` and `spdk_json_write_named_double` to allow 592for writing and decoding of the the double data type. 593 594### log 595 596Added consistent tracking of use of deprecated behavior. Deprecations continue to be 597listed in deprecation.md and now are in subsections with headers matching a tag 598used in `SPDK_LOG_DEPRECATED()` calls. When deprecated behavior is used, these 599tags will appear in SPDK's log at the warn level. As the SPDK application exits, it 600will log a summary of how many times `SPDK_LOG_DEPRECATED()` was called for each 601tag that was logged at least once. 602 603### nvme 604 605NVMe transport options were newly introduced. The NVMe transport options are defined via 606the `spdk_nvme_transport_opts` structure and configured via `spdk_nvme_transport_get_opts` 607and `spdk_nvme_transport_get_opts` functions. 608 609Shared receive queue was supported by the RDMA transport. It can be configured by 610a new NVMe transport option `rdma_srq_size`. 611 612Added `transport_tos` to `spdk_nvme_ctrlr_opts` to support setting of the "type of service" 613value in the IPv4 header. Only RDMA is supported at this time. 614 615Added API `spdk_nvme_qpair_get_num_outstanding_reqs` to get the number of outstanding reqs of 616a specified qpair. 617 618### nvmf 619 620Added API `spdk_nvmf_tgt_pause_polling` and `spdk_nvmf_tgt_resume_polling` to allow 621pausing polling on poll group of a given target. 622 623The `num_shared_buffers` option now controls the maximum number of buffers a transport will 624pull from the central `iobuf` pool. It will not allocate additional memory. If the requested 625amount cannot be satisfied, a warning will be displayed to increase the size of the `iobuf` 626pool. 627 628### rpc 629 630Added `spdk_rpc_set_allowlist` to restrict allowed RPCs to the specified list. 631 632### scheduler 633 634Changing scheduler from dynamic back to static is no longer possible, 635as there is no way of saving original SPDK thread distribution on reactors. 636 637### thread 638 639Added `spdk_thread_get_app_thread` which returns the first thread that was created using 640`spdk_thread_create`. 641 642Added `spdk_thread_is_running`. This returns `true` for a running thread, or `false` if 643its exit process has started using `spdk_thread_exit`. 644 645Added API `spdk_spin_init`, `spdk_spin_destroy`, `spdk_spin_lock`, `spdk_spin_unlock`, and 646`spdk_spin_held` to support spinlocks that are aware of the SPDK concurrency model. 647 648Added iobuf buffer pool, set of API calls for allocating data buffers across libraries. 649Please see new API `spdk_iobuf_*`. 650 651### trace 652 653New `trace_get_info` RPC was added to get name of shared memory file, list of the 654available trace point groups and mask of the available trace points for each group. 655 656### ublk device 657 658The ublk application supports the ublk kernel driver. It's implemented as a ublk backend 659in spdk_tgt and could be started with specifying configuration. See the 660[ublk](https://www.kernel.org/doc/html/latest/block/ublk.html) documentation for more details. 661 662ublk bdev could export a block device via Linux ublk. It will move this backend device into userspace 663as `/dev/ublkb*`. Before to adding ublk device, need to create ublk target by RPC method. 664 665### util 666 667New API `spdk_fd_group_get_epoll_event` that returns the epoll(7) event that 668caused a function callback in file descriptor group to execute. 669 670Added API `spdk_strarray_from_string`, `spdk_strarray_dup` and `spdk_strarray_free` 671for handling arrays of strings. 672 673A new API `spdk_strcpy_replace` was added to replace all occurrences of the search string 674with the replacement string. 675 676New API `spdk_iov_memset()` was added to memset an iovec. 677 678New API `spdk_iov_one()` was added to initialize an iovec for a single buffer. 679 680## v22.09 681 682### accel 683 684Many names were changed in the accel framework to make them consistent both with themselves and 685the rest of SPDK. The primary public header file is now named `include/spdk/accel.h`. 686 687Added a new runtime RPC `accel_get_opc_assignments` to get a list of current opcode to engine 688assignments. 689 690Added a new startup RPC `accel_assign_opc` to assign/override a specific opcode to 691an engine. 692 693### bdev 694 695New RPCs `bdev_xnvme_create` and `bdev_xnvme_delete` were added to support the xNVMe bdev. 696 697A new API `spdk_bdev_for_each_bdev_io` was added to execute the function on the appropriate 698thread for each bdev_io submitted to the bdev. 699 700A new API `spdk_bdev_get_current_qd` was added to measure and return the queue depth from a 701bdev. This API is available even when queue depth sampling is disabled. 702 703Added `spdk_bdev_seek_data`, `spdk_bdev_seek_hole` and `spdk_bdev_io_get_seek_offset` functions 704that start from a given offset and seek for next data or for next hole. At this time only 705supported by lvol bdev module. 706 707A new parameter `io_drain_timeout` has been added to `spdk_bdev` structure. It controls how long 708a bdev reset must wait for IO to complete prior to issuing a reset to the underlying device. 709If there is no outstanding IO at the end of that period, the reset is skipped. 710Best defined for bdevs that share an underlying bdev, such as multiple lvol bdevs sharing 711an nvme device, to avoid unnecessarily resetting the underlying bdev and affecting other 712bdevs that are sharing it. 713Modifying this field is optional. Setting the value to 0 keeps the original behavior 714to always send resets immediately, even if there is no I/O outstanding. 715 716### blobstore 717 718Reserve space for `used_clusters` bitmap. The reserved space could be used for blobstore growing 719in the future. 720 721Added `is_zeroes` operation to `spdk_bs_dev`. It allows to detect if logical blocks are backed 722by zeroes device and do a shortcut in copy-on-write flow by excluding copy part from zeroes device. 723 724Added `spdk_blob_get_next_allocated_io_unit` and `spdk_blob_get_next_unallocated_io_unit` functions 725that start from a given offset and seek for first io_unit belonging to an allocated cluster 726or first io_unit belonging to an unallocated cluster. 727 728Added `spdk_bs_grow` function to grow blobstore size if the underlying bdev size is increased. 729This is used by lvol bdev via `bdev_lvol_grow_lvstore` RPC. 730 731### daos 732 733Added new DAOS bdev module, that creates SPDK block device on top of DAOS DFS. 734Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_daos) for more details. 735 736### env 737 738Added `spdk_pci_register_device_provider` and matching `SPDK_PCI_REGISTER_DEVICE_PROVIDER` 739functions for registering PCI device providers. That allow the VMD driver to be notified 740when users want to attach a device under a given BDF and when a device is detached. 741 742### init 743 744`spdk_subsystem_init_from_json_config` now fails if the JSON configuration file is not 745an object with an array named "subsystems". 746 747### iscsi 748 749Added `bdev_iscsi_set_options` RPC to modify options of the iSCSI bdev module. 750 751### json 752 753Added `spdk_json_find` API return errcode: EPROTOTYPE - json not enclosed in {}. 754`spdk_json_find` now returns -EPROTOTYPE instead of -ENOENT if the object parameter 755does not point to a JSON object (i.e. is not enclosed with {}). 756 757### lvol 758 759Add `num_md_pages_per_cluster_ratio` parameter to the `bdev_lvol_create_lvstore` RPC. 760Calculate `num_md_pages` from `num_md_pages_per_cluster_ratio`, and pass it to `spdk_bs_opts`. 761 762### nvme 763 764Added SPDK_NVME_TRANSPORT_CUSTOM_FABRICS to enum `spdk_nvme_transport_type` to support custom 765fabric transport. SPDK_NVME_TRANSPORT_CUSTOM was intended to be non-fabric custom transport. 766 767Added a new function `spdk_nvme_ns_cmd_verify` to submit a Verify Command to a Namespace. 768 769Added `spdk_nvme_ctrlr_disable_read_changed_ns_list_log_page` to allow an application to 770tell the driver to not read the CHANGED_NS_LIST log page in response to a NS_ATTR_CHANGED 771AEN. When called the application is required to read this log page instead to clear the 772AEN. 773 774Added `psk` field to `spdk_nvme_ctrlr_opts` struct in order to enable SSL socket implementation 775of TCP connection and set the PSK. Applicable for TCP transport only. 776 777### raid 778 779Renamed the `raid5` module to `raid5f` to reflect that it is not a traditional 780RAID5 implementation - only full stripe writes are supported, partial stripe 781writes (read-modify-write) are not. 782 783### rpc 784 785Added `psk` parameter to `bdev_nvme_attach_controller` RPC in order to enable SSL socket implementation 786of TCP connection and set the PSK. Applicable for TCP transport only. 787 788New options `enable_ktls` and `tls_version` were added to the `sock_impl_set_options` structure. 789New options `psk_key` and `psk_identity` were added to the `sock_impl_set_options` structure. 790 791Added warning message for `bdev_rbd_create`, if it is used without -c. 792`bdev_rbd_create()` API without specifying -c is deprecated and will be removed in future release. 793 794Renamed `enable_vmd` RPC to `vmd_enable` to make it consistent with our naming scheme of 795`<subsystem>_<action>`. For now, the old name is still available, but is marked as deprecated. 796 797New function `spdk_rpc_get_method_state_mask` was added to RPC library. 798 799### sma 800 801Extended `VolumeParameters` with crypto parameters allowing the user to configure crypto when 802attaching a volume to a device. This interface is now supported by all upstream device types 803(nvmf-tcp, virtio-blk, vfio-user) using `bdev_crypto`. Users must specify the crypto engine to use 804under `crypto` section in config. It is also possible to register out-of-tree crypto engines by 805inheriting from the `CryptoEngine` class. 806 807Added two new methods: `SetQos` and `GetQosCapabilities` allowing the user to configure QoS on a 808per-device or per-volume level. Not all QoS settings have to be supported by each device, so users 809can use `GetQosCapabilities` to query them for that. All upstream device types support QoS on a 810per-volume level using bdev layer's QoS mechanism. 811 812### sock 813 814Added new `ssl` based socket implementation, the code is located in `module/sock/posix`. 815For now we are using hard-coded PSK and only support TLS 1.3. 816 817### util 818 819Added new functions: `spdk_hexlify` and `spdk_unhexlify`. 820 821A new API `spdk_xor_gen` was added to generate XOR from multiple source buffers. It is going to be 822used by `raid5f` for calculating parity. 823 824### vfu_tgt 825 826Added vfu_tgt library abstraction based on libvfio-user for PCI device emulation. 827Besides the NVMe device emulation, its now possible to emulate virtio-blk and virtio-scsi 828devices. 829 830### virtio 831 832virtio-vhost-user no longer tries to support dynamic memory allocation. The vhost target does 833not support the high rate of SET_MEM_TABLE vhost messages that result from dynamic memory 834allocation, so a virtio-vhost-user device will now present an ERRLOG, assert, and skip the 835SET_MEM_TABLE vhost message if a memory notification is received outside of the normal device 836start/stop. Applications using the virtio library in vhost-user mode should now pre-allocate 837the application's memory using the -s/--mem-size option and use single shared memory file 838segments using the -g/--single-file-segments option. 839 840Added `vfio-user` transport type for virtio-blk and virtio-scsi devices. 841 842### vmd 843 844Fixed hotplug when a device is inserted in a slot in which a disk was already enumerated previously. 845Added two new RPCs: `vmd_remove_device` simulating a hotremove, and `vmd_rescan`, which rescans all 846buses managed by the VMD driver and hotplugs all newfound devices. 847 848## v22.05 849 850### sock 851 852A new option `ack_timeout` was added to the `spdk_sock_opts` structure. 853 854### util 855 856A new parameter `bounce_iovcnt` was added to `spdk_dif_generate_copy` and `spdk_dif_verify_copy`. 857The `bounce_iovcnt` is used to specify the number of bounce_iov to support multiple block-aligned 858fragment copies. 859 860A new API `spdk_copy_iovs_to_buf` and `spdk_copy_buf_to_iovs` were added to copy iovs to buf or 861copy buf to iovs. There're many cases need to use these two APIs. 862 863### bdev 864 865Removed deprecated spdk_bdev_module_finish_done(). Use spdk_bdev_module_fini_done() instead. 866 867A new API `spdk_bdev_unregister_by_name` was added to handle race conditions correctly. 868 869New APIs, `spdk_for_each_bdev` and `spdk_for_each_bdev_leaf`, were added to provide iteration 870safe for race conditions. 871 872A new RPC `bdev_nvme_get_io_paths` was added to get all active I/O paths. 873 874A new RPC `bdev_nvme_set_preferred_path` was added to set preferred I/O path for an NVMe bdev 875when in multipath mode. This RPC does not support NVMe bdevs in failover mode. 876 877A new RPC `bdev_nvme_set_multipath_policy` was added to set multipath policy of a NVMe bdev 878in multipath mode. 879 880A new option `disable_auto_failback` was added to the `bdev_nvme_set_options` RPC to disable 881automatic failback. 882 883### idxd / dsa 884 885A new parameter `flags` was added to all low level submission and preparation 886APIs to enable the caller to pass operation flags per the DSA specification. 887 888A new flag 'SPDK_IDXD_FLAG_PERSISTENT' was added to let DSA know that 889the destination is persistent. 890 891The RPC `idxd_scan_accel_engine` has been renamed to `dsa_scan_accel_engine` 892 893The RPC `iaa_scan_accel_engine` has been added. 894 895Many HW related structs/functions with the name `idxd` have been renamed `dsa` 896to more accurately represent the HW they are associated with. 897 898Two new functions were added to the library `spdk_idxd_submit_compress` and 899`spdk_idxd_submit_decompress` 900 901### accel_fw 902 903A new parameter `flags` was added to accel API. 904The APIs include: 905`spdk_accel_submit_copy` 906`spdk_accel_submit_dualcast` 907`spdk_accel_submit_fill` 908`spdk_accel_submit_copy_crc32c` 909`spdk_accel_submit_copy_crc32cv` 910`spdk_accel_submit_compress` 911`spdk_accel_submit_decompress` 912 913A new flag `ACCEL_FLAG_PERSISTENT` was added to indicate the target memory is PMEM. 914 915The API `spdk_accel_get_capabilities` has been removed. 916 917### crypto 918 919Support for AES_XTS was added for MLX5 polled mode driver (pmd). 920 921bdev_crypto_create RPC now requires hexlified 'key' and 'key2' params for all pmd drivers. 922Unhexlifying is performed during RPC command processing and the vbdev crypto module runs on 923binary keys as before. 924 925### bdev 926 927Added a timeout option to the `bdev_get_bdevs` RPC. It allows the user to specify the amount of 928time to wait until a bdev with a given name appears in the system. 929 930### bdev_nvme 931 932Added `bdev_nvme_add_error_injection` and `bdev_nvme_remove_error_injection` RPCs to add and 933remove NVMe error injections. 934 935New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 936added to the RPC `bdev_nvme_set_options`. They can be overridden if they are given by the RPC 937`bdev_nvme_attach_controller`. 938 939### blobstore 940 941New functions `spdk_blob_io_writev_ext` and `spdk_blob_io_readv_ext` are added. The new functions accept 942`spdk_blob_ext_io_opts` structure with extended IO request options. 943 944### event 945 946Added `msg_mempool_size` parameter to `spdk_reactors_init` and `spdk_thread_lib_init_ext`. 947The size of `g_spdk_msg_mempool` can now be controlled through the same-named 948user option of `spdk_app_opts` structure. 949 950### nvme 951 952The API `spdk_nvme_ctrlr_prepare_for_reset()` was deprecated. The functionality provided by the 953`spdk_nvme_ctrlr_prepare_for_reset()` was merged into the API `spdk_nvme_ctrlr_disconnect()`. 954 955### nvmf 956 957Removed deprecated max_qpairs_per_ctrlr parameter from nvmf_create_transport RPC. Use 958max_io_qpairs_per_ctrlr instead. 959 960Deprecated the ability for hosts to connect to the discovery subsystem automatically on any 961existing listener. Users should now explicitly add listeners for the discovery subsystem. 962Host can still connect to the discovery subsystem as before, but a warning message will be 963emitted if no listener was configured for the transport ID of the incoming connection. 964 965Added adaptive interrupt feature for vfio-user transport. New parameter `disable_adaptive_irq` 966is added to the RPC `nvmf_create_transport`. 967 968### thread 969 970Added `spdk_thread_exec_msg()` API. 971 972### scheduler 973 974`framework_set_scheduler` can now be called after application initialization. 975Added callbacks to set custom parameters specific for each scheduler implementation 976and `framework_get_scheduler` to retrieve them. 977 978Added `dynamic` scheduler options: load_limit, core_limit, core_busy. Their descriptions 979are available in JSON-RPC document, in section 980[framework_set_scheduler](jsonrpc.html#rpc_framework_set_scheduler). 981 982### raid 983 984Add concat as a special raid module. The concat module could create a virtual bdev. The 985virtual bdev combines multiple underlying bdevs together. The layout of the underlying 986bdevs is one after another. The concat bdev is extendable. When the free space of the 987concat bdev is not enough, the user can deconstruct the concat bdev, then reconstruct it 988with an additional underlying bdev. 989 990### sock 991 992Allow MSG_ZEROCOPY flag to be set or not according to data size, which can be enabled and 993set by setting "zerocopy_threshold". zerocopy_threshold = 0 means disable this function; 994zerocopy_threshold > 0 means enable it and use this value as the threshold. 995 996### rpc 997 998Introduced `zerocopy_threshold` to enable zerocopy on send for server sockets according to 999data size to be flushed. 1000 1001## v22.01 1002 1003### accel 1004 1005The batching capability was removed. Batching is now considered an implementation 1006detail of the low level drivers. 1007 1008### bdev 1009 1010The NVMe bdev module supports multipath and improved I/O error resiliency. 1011 1012The parameter `retry_count` of the RPC `bdev_nvme_set_options` was deprecated and will be 1013removed in SPDK 22.04, and the parameter `transport_retry_count` is added and used instead. 1014 1015An new parameter `bdev_retry_count` is added to the RPC `bdev_nvme_set_options`. 1016 1017New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 1018added to the RPC `bdev_nvme_attach_controller`. 1019 1020An new parameter `num_io_queues` is added to `bdev_nvme_attach_controller` RPC to allow specifying amount 1021of requested IO queues. 1022 1023Added `key_file` parameter to the `rbd_register_cluster` RPC. It is an optional parameter to 1024specify a keyring file to connect to a RADOS cluster. 1025 1026Added `spdk_bdev_get_zone_id` API to retrieve zone_id for a certain LBA. 1027 1028### bdev_aio 1029 1030Added `bdev_aio_rescan` RPC to allow rescanning the size of aio bdev. 1031 1032### bdev_nvme 1033 1034Added discovery service to bdev_nvme module, which now can connect to a discovery controller. 1035bdev_nvme will connect to all current and future subsystems in the discovery controller. 1036Added `bdev_nvme_start_discovery` and `bdev_nvme_stop_discovery` RPC. 1037 1038### dpdk 1039 1040Updated DPDK submodule to DPDK 21.11. 1041 1042### env 1043 1044Added `spdk_pci_for_each_device`. 1045 1046Removed `spdk_pci_get_first_device` and `spdk_pci_get_next_device`. These APIs were unsafe, because 1047they did not account for PCI devices being inserted or removed while the caller was using handles 1048returned from these APIs. Existing users of these APIs should switch to `spdk_pci_for_each_device` 1049instead. 1050 1051Added 3 experimental APIs to handle PCI device interrupts (`spdk_pci_device_enable_interrupt`, 1052`spdk_pci_device_disable_interrupt`, `spdk_pci_device_get_interrupt_efd`). 1053 1054Added `framework_get_pci_devices` RPC to list PCIe devices attached to an SPDK application. 1055 1056### idxd 1057 1058Many APIs are now vectored rather than scalar, meaning they take iovecs instead of individual pointers. 1059 1060### json 1061 1062Added `spdk_json_write_bytearray` API to serialize a buffer as a hex string. 1063 1064### nvme 1065 1066API `spdk_nvme_trtype_is_fabrics` was added to return existing transport type 1067is fabric or not. 1068 1069API `spdk_nvme_poll_group_remove` was limited to be available only for a 1070disconnected qpair in the group. 1071 1072New APIs, `spdk_nvme_ctrlr_disconnect`, `spdk_nvme_ctrlr_reconnect_async`, and 1073`spdk_nvme_ctrlr_reconnect_poll_async`, have been added to improve error recovery, and 1074the existing APIs,`spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async` 1075were deprecated. 1076 1077Added `spdk_nvme_ctrlr_get_discovery_log_page` API for getting the full discovery log page 1078from a discovery controller. 1079 1080### nvmf 1081 1082Added support for zero-copy operations in the NVMe-oF TCP target. It can be enabled via 1083the `zcopy` parameter when creating a transport. The zero-copy operations are only used 1084for requests not using in-capsule data. 1085 1086Added a `subsystem` parameter to `spdk_nvmf_transport_stop_listen_async`. When not NULL, 1087it will only disconnect qpairs for controllers associated with the specified subsystem. 1088 1089Removed accept poller from transport layer. Each transport can have its own policy of 1090handling new connections. To notify transport layer `spdk_nvmf_poll_group_add` and 1091`spdk_nvmf_tgt_new_qpair` can be used. 1092 1093### scsi 1094 1095Structure `spdk_scsi_lun` has been extended with new member `resizing` so that SCSI layer now reports 1096unit attention for disk resize. 1097 1098### trace 1099 1100Added `spdk_trace_create_tpoint_group_mask` to return tracepoint group mask from 1101a tracepoint group name. 1102 1103Added `trace_set_tpoint_mask` and `trace_clear_tpoint_mask` RPC to allow enabling 1104individual traces. 1105 1106### util 1107 1108Added `spdk_ioviter_first` and `spdk_ioviter_next` to iterate over two iovecs and 1109yield pointers to common length segments. 1110 1111### sock 1112 1113A new parameter `hint` is added to `spdk_sock_get_optimal_sock_group`. It allows to suggest 1114a poll group when no optimal poll group is found. 1115 1116## v21.10 1117 1118Structure `spdk_nvmf_target_opts` has been extended with new member `discovery_filter` which allows to specify 1119filtering rules applied during discovery log generation. Refer to `enum spdk_nvmf_tgt_discovery_filter` for more info. 1120 1121### bdev 1122 1123New API `spdk_bdev_get_memory_domains` has been added, it allows to get SPDK memory domains used by bdev. 1124 1125New API functions `spdk_bdev_readv_blocks_ext` and `spdk_bdev_writev_blocks_ext` have been added. 1126These functions accept `spdk_bdev_ext_io_opts` structure with extended IO request 1127options, e.g. DMA memory domain which describes data that may belong to another memory domain and 1128can't be accessed directly. 1129 1130Added `async_fini_start` to allow bdev modules to complete the `fini_start` asynchronously, 1131with new `spdk_bdev_module_fini_start_done` API. 1132 1133Deprecated `spdk_bdev_module_finish_done()` API, which will be removed in SPDK 22.01. 1134Bdev modules should use `spdk_bdev_module_fini_done()` instead. 1135 1136The `ocssd` bdev has been removed. The Open Channel specification has been largely superseded by 1137zoned namespaces, few if any devices have been brought to market, and there is little reason to 1138continue to support this. OCSSD support in the nvme driver will remain for now. 1139 1140### dma 1141 1142A new library, lib/dma, has been added. This library provides the necessary infrastructure for 1143handling systems and devices with multiple memory domains. For example, a PCIe add-in card with an 1144SoC may be running SPDK on the SoC. That SoC has its own local memory, but SPDK may be controlling 1145devices that can also access the host system memory. This library provides infrastructure to enumerate 1146the memory domains and request hardware perform DMA transfers between them. 1147 1148### dpdk 1149 1150Updated DPDK submodule to DPDK 21.08. 1151 1152### event 1153 1154Added the `disable_signal_handlers` flag to the `spdk_app_opts` struct. 1155 1156### idxd 1157 1158Added `spdk_idxd_get_socket` to query the socket that the idxd device 1159is on. 1160 1161### json 1162 1163Added API to allow for writing and decoding of new types: 1164 1165- `spdk_json_write_named_uint16` 1166- `spdk_json_write_named_uint8` 1167- `spdk_json_write_uint16` 1168- `spdk_json_write_uint8` 1169- `spdk_json_number_to_uint8` 1170- `spdk_json_decode_uint8` 1171 1172### log 1173 1174Added API `spdk_log_to_syslog_level` to return syslog level based on SPDK's 1175log level. 1176 1177### nvme 1178 1179Added new functions `spdk_nvme_zns_set_zone_desc_ext` and `spdk_nvme_zns_ext_report_zones` 1180to set zone descriptor extension and to get extended zone report respectively. 1181 1182New API `spdk_nvme_ctrlr_get_memory_domains` has been added, it allows to get SPDK memory domains used by nvme controller. 1183 1184New API functions `spdk_nvme_ns_cmd_readv_ext` and `spdk_nvme_ns_cmd_writev_ext` 1185have been added. These functions accept `spdk_nvme_ns_cmd_ext_io_opts` structure with extended IO request 1186options, e.g. DMA memory domain which describes data that may belong to another memory domain and 1187can't be accessed directly. 1188 1189Added a new function `spdk_nvme_ctrlr_get_regs_bpinfo` to get boot partition info of a controller. 1190Added new functions `spdk_nvme_ctrlr_write_boot_partition`, 1191`spdk_nvme_ctrlr_read_boot_partition_start` and `spdk_nvme_ctrlr_read_boot_partition_poll` 1192to write and read the boot partitions of a controller. 1193 1194Added `spdk_nvme_ctrlr_get_opts` to retrieve the current controller options. 1195 1196Added `async_mode` to `spdk_nvme_io_qpair_opts` to enable creation of submission and completion 1197queues asynchronously. This mode is currently supported at PCIe layer, 1198which tracks the qpair creation with state machine and returns to the user immediately. 1199Default mode is set to false to create io qpairs synchronously. 1200 1201Added `spdk_nvme_ctrlr_get_regs_cc` to retrieve NVMe controller CC (Configuration) register. 1202 1203Added `spdk_nvme_ctrlr_prepare_for_reset` to inform the driver that the application is preparing 1204to reset the specified NVMe controller. This function allows the driver to make decisions 1205knowing that a reset is about to happen. 1206 1207### nvmf 1208 1209Added `oncs` to `struct spdk_nvmf_ctrlr_data` so that the transport layer 1210can decide support RESERVATION feature or not. 1211 1212An `opts_size` element was added in the `spdk_nvmf_ns_opts` structure to solve the 1213ABI compatibility issue between different SPDK version. An new option `anagrpid` was 1214added in the `spdk_nvmf_ns_opts` structure. 1215 1216An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_add_ns`. 1217 1218An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_listener_set_ana_state`. 1219 1220`spdk_nvmf_subsystem_destroy` is now can be asynchronous, it accepts a callback and callback argument. 1221 1222A new parameter, `poll_groups_mask` was added to the `nvmf_set_config` RPC that allows specifying 1223a subset of cores for the nvmf poll groups. This helps to avoid imbalances when some cores are 1224busy with periodic timer tasks that run very frequently. 1225 1226A new parameter, `commit` was added to `nvmf_bdev_ctrlr_end_zcopy` that allows indicating whether 1227the buffers should be committed. 1228 1229Added `listen_dump_opts` to `spdk_nvmf_transport_ops` so that the transport layer can display 1230listen opts. 1231 1232Added `fabrics_connect_timeout_us` to `bdev_nvme_attach_controller` RPC to allow specifying 1233timeout for connect operation. 1234 1235Added `multipath` to `bdev_nvme_attach_controller` RPC to allow specifying multipath behavior. 1236 1237### scheduler 1238 1239New API for implementing schedulers and governors. Please see `include/spdk/scheduler.h` for details. 1240 1241### scsi 1242 1243New functions, `spdk_scsi_dev_get_first_lun` and `spdk_scsi_dev_get_next_lun` 1244have been added to iterate LUNs of a SCSI device. 1245 1246Each SCSI device supports 256 LUNs at the maximum now and the macro constant 1247`SPDK_SCSI_DEV_MAX_LUN` was removed. 1248 1249### trace_parser 1250 1251A new library, lib/trace_parser, has been added. This library provides functions that parse traces 1252recorded by an SPDK application. That includes merging traces from multiple cores, 1253sorting them by their timestamp and constructing trace entries spanning across multiple buffers. 1254 1255### util 1256 1257The `spdk_fd_group_add` API now takes a `name` parameter. 1258 1259## v21.07 1260 1261### accel_fw 1262 1263Added API `spdk_accel_submit_copy_crc32c` to perform a CRC32C while copying data. 1264 1265Added API `spdk_accel_batch_prep_copy_crc32c` to batch CRC32C + copy commands. 1266 1267Added API `spdk_accel_submit_copy_crc32cv` to submit chained CRC32C + copy commands. 1268 1269### bdev 1270 1271Change `spdk_bdev_read_blocks_with_md` arg offset definition from int64_t to uint64_t. 1272 1273Red-black tree has been used to organize the bdev names and aliases uniformly 1274to provide faster lookup. 1275 1276Removed ZCOPY emulation: The bdev module can be checked to see if it supports ZCOPY 1277and if not supported then use existing READ/WRITE commands. 1278 1279Added iov to spdk_bdev_zcopy_start. 1280 1281### dpdk 1282 1283Updated DPDK submodule to DPDK 21.05. 1284 1285### idxd 1286 1287Remove the `probe_cb` parameter in `spdk_idxd_probe` function. And remove the definition 1288of `spdk_idxd_probe_cb` function pointer. It should be implemented in idxd_user.c. 1289 1290Added API `spdk_idxd_submit_copy_crc32c` to perform a CRC32C while copying data. 1291 1292Added API `spdk_idxd_batch_prep_copy_crc32c` to prepare a batch operation to perform 1293a CRC32C while copying data. 1294 1295We have userspace idxd driver under accel_engine library (module/accel/idxd/accel_engine). 1296And kernel idxd driver usage is also added under the accel_engine library. This approach 1297implemented a wrapper library to use IDXD device by leveraging the kernel DSA driver in 1298SPDK idxd library (lib/idxd). Then users can leverage the RPC later to configure how to 1299use the DSA device by user space driver or kernel driver. 1300 1301### init 1302 1303Added new `init` library that initializes the SPDK subsystems, which previously was 1304internal to application framework. That functionality has been made public and for 1305use by applications which don't leverage SPDK's application framework. 1306 1307### iscsi 1308 1309New parameters, `pdu_pool_size`, `immediate_data_pool_size`, and `data_out_pool_size`, 1310were added to the RPC `iscsi_set_options` to run iSCSI target with varying amount of 1311available memory. 1312 1313### json 1314 1315Added API `spdk_json_write_named_uint128` and `spdk_json_write_uint128` to perform 1316the uint128 related data. 1317 1318### net 1319 1320Removed deprecated `net` library. 1321 1322Added support for ZCOPY. ZCOPY is used in preference to READ and WRITE if the 1323bdev module supports ZCOPY and the bdev module has ZCOPY enabled. 1324 1325### nvme 1326 1327`spdk_nvme_map_prps` and `spdk_nvme_map_cmd` were moved to nvmf/vfio-user as internal APIs 1328as vfio-user is the only user for the above two APIs. 1329 1330Added a new function `spdk_nvme_ns_cmd_copy` to submit a Simple Copy Command to a Namespace. 1331 1332Update the `spdk_nvme_generic_command_status_code` structure with new status code 1333according to the definition in NVMe 1.4 spec. 1334 1335`spdk_nvme_ctrlr_get_default_ctrlr_opts` now sets `use_cmb_sqs` to false. This means 1336that if a controller has a CMB and supports SQs in the CMB, SPDK will not use 1337the CMB for SQs by default - the user must set `use_cmb_sqs` to true in 1338the `spdk_nvme_ctrlr_opts` structure prior to controller attach. 1339 1340Add a new function `spdk_nvme_detach_poll` to simplify a common use case to continue 1341polling until all detachments complete. 1342 1343Added new argument `timeout_admin_us` to `spdk_nvme_ctrlr_register_timeout_callback` so callers 1344can specify a different timeout for admin commands vs. io commands. 1345 1346An existing function `spdk_nvme_detach_async` was updated to add one or more detachments 1347to an active context while it is being polled. 1348 1349Rename a variable in the member `cmic` of the struct `spdk_nvme_ctrlr_data` from 1350`multi_host` to `multi_ctrlr`. The variable means that the NVM subsystem may have two or 1351more controllers if set to 1. However `multi_host` had indicated a particular use case 1352such that the NVM subsystem is used by multiple hosts. 1353 1354A new option `disable_read_ana_log_page` was added to struct `spdk_nvme_ctrlr_opts` to disable 1355reading ANA log page. The upper layer is expected to read ANA log page instead if `true`. 1356The default value is `false`. 1357 1358New APIs, `spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async`, have been added to 1359reset a controller asynchronously. 1360 1361New RPC `bdev_nvme_reset_controller` was added, to reset an NVMe controller. 1362 1363Added `spdk_nvme_ns_get_nguid` function to get NGUID for the given namespace. 1364 1365Added `spdk_nvme_ctrlr_is_fabrics` function to indicate whether a ctrlr handle 1366is associated with a fabrics controller. 1367 1368### nvmf 1369 1370Added `min_cntlid` and `max_cntlid` to `nvmf_create_subsystem` to limit the controller ID range. 1371Added `spdk_nvmf_subsystem_get_min_cntlid` and `spdk_nvmf_subsystem_get_max_cntlid` to request those values. 1372 1373`spdk_nvmf_request_get_buffers_multi` API is removed. 1374 1375Added the `nvmf_set_crdt` RPC for setting command retry delay times. 1376 1377Expanded `spdk_nvmf_poll_group_stat` with current qpair count statistics. 1378 1379Removed following deprecated APIs: 1380 1381- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 1382- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 1383- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 1384- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 1385- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 1386- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 1387- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 1388 1389### rpc 1390 1391New RPC `bdev_rbd_register_cluster` and `bdev_rbd_unregister_cluster` was added, it allows to create 1392and delete the rados object cluster, then users can choose the cluster to create related rbd 1393device. 1394 1395Revised `bdev_rbd_create` parameter, it allows to use an optional parameter `--cluster-name` 1396to create a rbd bdev with an already registered Rados Cluster Object. 1397 1398New RPC `bdev_rbd_get_clusters_info` was added, it allows to get the info of the registered 1399Rados Cluster names. 1400 1401New optional parameter, `timeout_admin_us`, added to the bdev_nvme_set_options RPC. 1402 1403Revised a parameter `--stripe-size_kb` to `--stripe-size-kb` of `bdev_raid_create` method 1404provided in `scripts/rpc.py` for consistency. 1405 1406An new optional parameter `config_kernel_mode` was added to the RPC `idxd_scan_accel_engine`, 1407and this is used to enable using the kernel mode IDXD driver. 1408 1409Removed deprecated parameter `enable-zerocopy-send` of RPC `sock_impl_set_options`, 1410use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 1411Removed deprecated parameter `disable-zerocopy-send` of RPC `sock_impl_set_options`, 1412use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 1413 1414### rpm 1415 1416Removed deprecated `pkg/spdk.spec`, please use `rpmbuild/spdk.spec` instead. 1417See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 1418 1419### thread 1420 1421Red-black tree has been used for timed pollers to provide faster insertion and deletion 1422and for io_devices to provide faster lookup. 1423 1424Removed `spdk_io_channel` structure from public header and moved it to thread_internal.h. 1425 1426### trace 1427 1428Added `spdk_trace_register_description_ext` function to register variable number of tracepoint arguments. 1429 1430Added ability to chain multiple trace entries together to extend the size of the argument buffer. 1431 1432### util 1433 1434Red-black tree macros has been added by using the macros provided by the FreeBSD operating system 1435under the same BSD license. 1436 1437Add an new macro `SPDK_SIZEOF_MEMBER` to get the size of a member of a struct. 1438 1439`spdk_crc32c_iov_update` function was added to support calculating the crc32c of the iovs. 1440 1441Added zipf random number generator with power law probability distribution. 1442When applied to performance testing of block devices, it will select blocks over 1443the full range of LBAs, but will more frequently select lower-numbered LBAs. 1444 1445## v21.04 1446 1447### accel 1448 1449Two new accelerated crc32 functions `spdk_accel_submit_crc32cv` and 1450`spdk_accel_batch_prep_crc32cv` are added in order to provide the 1451chained accelerated CRC32 computation support. 1452 1453### bdev 1454 1455For `bdev_ocssd_create` RPC, the optional parameter `range` was removed. 1456Only one OCSSD bdev can be created for one OCSSD namespace. 1457 1458Removed the `spdk_bdev_open` from bdev library API. 1459Removed the `spdk_vbdev_register` and `spdk_bdev_part_base_construct` from bdev module API. 1460Removed the `config_text` function for bdev modules to report legacy config. 1461 1462Added `spdk_bdev_get_max_active_zones` API to display maximum number active zones of a given bdev. 1463 1464Added `spdk_bdev_get_max_zone_append_size` API to display maximum zone append data transfer size. 1465 1466### bdev_nvme 1467 1468Added support for zoned namespaces. 1469 1470### blobstore 1471 1472Removed the `spdk_bdev_create_bs_dev_from_desc` and `spdk_bdev_create_bs_dev` API. 1473 1474### env 1475 1476Added `spdk_pci_device_allow` API to allow applications to add PCI addresses to 1477the allowed list after the application has started. 1478 1479Removed the `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts`. 1480 1481Added hotplug support based on uevent in `pci_event.c`. Added usage of this functionality in 1482nvme, virtio-scsi and virtio_blk libraries. Please see the new API `spdk_pci_event_listen`, 1483`spdk_pci_get_event`, `spdk_pci_register_error_handler` and `spdk_pci_unregister_error_handler`. 1484 1485### event 1486 1487Removed the `config_file`, `max_delay_us`, `pci_whitelist` 1488and `pci_blacklist` members of struct `spdk_app_opts`. 1489 1490### idxd 1491 1492A new API `spdk_idxd_device_needs_rebalance` was added so that users of the library 1493can know whether they need to rebalance the flow control for the channel 1494that was just added/removed. This is based on how the low level library 1495shares devices amongst channels. 1496 1497The API `spdk_idxd_reconfigure_chan` had the `num_channels` removed as this 1498is now tracked in the library. The app makes use the new API above to 1499determine whether to rebalance or not. This applies to `spdk_idxd_configure_chan` 1500as well. 1501 1502The API `spdk_idxd_put_channel` now returns the rebalance state for the 1503underlying device. 1504 1505### iscsi 1506 1507A security vulnerability has been identified and fixed in the SPDK iSCSI target. 1508A TEXT PDU with no data, but CONTINUE flag set, would result in a NULL pointer dereference 1509and crash the SPDK iSCSI target process. All users of the SPDK iSCSI target 1510are recommended to update. All SPDK versions <= v21.01 are affected. 1511 1512### net 1513 1514The net library is deprecated and will be removed in the 21.07 release. 1515 1516### nvme 1517 1518Added a new function `spdk_nvme_ctrlr_get_regs_pmrcap` to get the PMR capabilities. 1519 1520Added an accelerated table pointer in `spdk_nvme_poll_group` 1521which can be used to provide the accelerated functions by users with 1522hardware engine, such as crc32c accelerated function. 1523 1524Added new functions `spdk_nvme_ctrlr_get_pmrsz`, `spdk_nvme_ctrlr_enable_pmr`, 1525`spdk_nvme_ctrlr_disable_pmr`, `spdk_nvme_ctrlr_map_pmr` and `spdk_nvme_ctrlr_unmap_pmr`. 1526 1527Added NVMe transport operations to enable, disable, map and unmap the PMR. 1528 1529Added `spdk_nvme_qpair_get_optimal_poll_group` function and `qpair_get_optimal_poll_group` 1530function pointer in spdk_nvmf_transport_ops structure in order to add the qpair to the most 1531suitable polling group. 1532 1533Added OPTPERF and namespace optimal performance fields to nvme_spec.h. 1534 1535Added `spdk_nvme_set_hotplug_filter` API to allow applications to choose which 1536hot-inserted SSDs should be probed. This is useful for use cases where multiple 1537independent SPDK processes are running on one node. The filter function can 1538then be implemented in these processes to decide which SSDs to probe based on 1539the new SSD's PCI address. 1540 1541New functions `spdk_nvme_poll_group_get_stats` and `spdk_nvme_poll_group_free_stats` 1542were added. These functions allow to get transport statistics per NVME poll group. 1543 1544Added `spdk_nvme_map_cmd` API to map the NVMe command with SGL cases. 1545 1546Added support for vector variant of ZNS zone append commands with new API 1547`spdk_nvme_zns_zone_appendv` and `spdk_nvme_zns_zone_appendv_with_md`. 1548 1549Added `spdk_nvme_zns_ns_get_max_open_zones` and `spdk_nvme_zns_ns_get_max_active_zones` API, 1550to display maximum number of open and active zones of the given namespace. 1551 1552Added `spdk_nvme_zns_ns_get_zone_size_sectors` API to provide size of zone in number of 1553sectors. 1554 1555Added `spdk_nvme_qpair_get_id` API to display the ID of the specified qpair. 1556 1557### nvmf 1558 1559Removed the `spdk_nvmf_tgt_listen` and `spdk_nvmf_subsystem_add_ns` API. 1560 1561Added new APIs: 1562 1563- `spdk_nvmf_poll_group_dump_stat` (function in `nvmf.h`). 1564- `poll_group_dump_stat` (transport op in `nvmf_transport.h`). 1565 1566The following APIs have been deprecated and will be removed in SPDK 21.07: 1567 1568- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 1569- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 1570- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 1571- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 1572- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 1573- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 1574- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 1575 1576See header files for details. 1577 1578The `trtype` field in JSON returned by `nvmf_get_stats` RPC contains now the name of the transport, 1579which is the same as the type for defined transports and more informative for a custom transport. 1580 1581Added `hdgst` and `ddgst` parameters to `bdev_nvme_attach_controller` RPC in order change 1582state of TCP header and data digest. 1583 1584Added `num_cqe` parameter to `nvmf_create_transport` RPC to set number of completion queues (CQ) 1585for RDMA transport. Useful when CQ resize operation is not supported, for example iWARP. 1586 1587### ocf 1588 1589Updated OCF submodule to v20.12.2 1590 1591Added `bdev_ocf_set_cache_mode` RPC to dynamically switch cache mode of OCF bdev. 1592 1593OCF deprecation notice has removed as Huawei is picking up support for the OCF project. 1594 1595### opal 1596 1597Removed the `spdk_opal_supported` API. 1598 1599### raid 1600 1601For `bdev_raid_create` RPC, the deprecated parameter `strip_size` was removed. 1602 1603### rpc 1604 1605New RPC `bdev_nvme_get_transport_statistics` was added, it allows to get transport statistics 1606of nvme poll groups. 1607 1608Parameter `enable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1609use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 1610Parameter `disable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1611use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 1612 1613Added cmd_parser.py used to parse the args from argparse. There are 1614two benefit to use command parser: 1615 1616- Simplify the definition of rpc method. It will reduce the rpc method code. 1617- Make the rpc call more versatile. User can add private args into rpc method. 1618 1619### rpm 1620 1621Added support for new RPM spec, rpmbuild/spdk.spec, which can be used for packaging the 1622SPDK. The pkg/spdk.spec is considered to be deprecated and scheduled for removal in SPDK 21.07. 1623See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 1624 1625### sock 1626 1627The type of `enable_placement_id` in struct `spdk_sock_impl_opts` is changed from 1628bool to int. We can use RPC to configure different value of `enable_placement_id`. 1629Then we can leverage SO_INCOMING_CPU to get placement_id, which aims to utilize 1630CPU cache locality, enabled by setting enable_placement_id=2. 1631 1632A new socket placement mode called PLACEMENT_MARK has been added. Some NICs allow 1633sockets to be marked using the SO_MARK socket option as a hint for which hardware 1634queue they should be associated with. This mode leverages that by setting the same 1635value for all sockets within a poll group. 1636 1637New parameters `enable_zerocopy_send_server` and `enable_zerocopy_send_client` were added 1638to struct spdk_sock_impl_opts, these parameters enable or disable zero copy send for server 1639and client sockets which are created using `spdk_sock_listen` and `spdk_sock_listen_ext` (server); 1640`spdk_sock_connect` and `spdk_sock_connect_ext` (client) functions. Existing parameter 1641`enable_zerocopy_send` enables or disables zero copy send for both server and client sockets. 1642 1643### thread 1644 1645A new API `spdk_io_channel_get_io_device` was added to get the io_device for the specified 1646I/O channel. 1647 1648Added `spdk_thread_set_interrupt_mode` function in order to set present spdk_thread into 1649interrupt mode or back to poll mode. It is valid only when thread interrupt facility is 1650enabled by `spdk_interrupt_mode_enable`. 1651 1652Added `spdk_poller_register_interrupt` function to mark that the poller is capable of 1653entering interrupt mode. Callback function will be called when the poller must transition 1654into or out of interrupt mode. 1655 1656### virtio 1657 1658Added the `bdev_virtio_blk_set_hotplug` RPC for the virtio blk pci device. 1659 1660## v21.01 1661 1662### bdev 1663 1664An `opts_size` element was added in the `spdk_bdev_opts` structure to solve the 1665ABI compatibility issue between different SPDK version. And also add `opts_size` 1666parameter in spdk_bdev_get_opts function. Two fields `small_buf_pool_size` and 1667`large_buf_pool_size` were added into spdk_bdev_opts, which were used to determine 1668the small and large buffer pool size of the whole bdev module. 1669 1670A new API `spdk_bdev_wait_for_examine` was added to allow for checking state of 1671examine process. Along with corresponding `bdev_wait_for_examine` RPC, which 1672is now always called during `spdk_bdev_subsystem_config_json` making sure 1673every bdev is ready to be used. 1674 1675A new API `spdk_bdev_io_get_aio_status` was added for getting the status of 1676bdev_io as Linux AIO errno. Also `spdk_bdev_io_complete_aio_status` function 1677and `SPDK_BDEV_IO_STATUS_AIO_ERROR` were added for bdev module to complete 1678a bdev_io with Linux AIO errno. 1679 1680A new API `spdk_bdev_get_module_name` was added for getting block device module name. 1681Also `spdk_bdev_get_module_ctx` was added to obtain a bdev module context for 1682the block device opened by the specified descriptor. 1683 1684Added `max_segment_size` and `max_num_segments` to `spdk_bdev` structure for 1685bdev modules to specify splitting requirements. 1686 1687### blob 1688 1689An `opts_size` element was added in the `spdk_bs_opts` structure to solve the 1690ABI compatibility issue between different SPDK version. And also add `opts_size` 1691parameter in `spdk_bs_opts_init` function. 1692 1693An `opts_size` element was added in the `spdk_blob_opts` structure to solve the 1694ABI compatibility issue between different SPDK version. And also add `opts_size` 1695parameter in `spdk_blob_opts_init` function. 1696 1697An `opts_size` element was added in the `spdk_blob_open_opts` structure to solve the 1698ABI compatibility issue between different SPDK version. And also add `opts_size` 1699parameter in `spdk_blob_open_opts_init` function. 1700 1701### build 1702 1703SPDK now generates pkg-config files to simplify the process of determining which 1704libraries must be linked into an SPDK application. 1705 1706### dpdk 1707 1708Updated DPDK submodule to DPDK 20.11. 1709 1710Removed `--with-igb-uio-driver` configure option. Since DPDK 20.11 igb_uio driver 1711was moved to separate dpdk-kmods repository. If required, users need to refer to 1712dpdk-kmods repository to build the driver. 1713 1714### env 1715 1716The `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts` 1717have been deprecated. The new members are named `pci_allowed`, `pci_blocked` and 1718`main_core` respectively. 1719 1720An `opts_size` element was added in the `spdk_app_opts` structure 1721to solve the ABI compatibility issue between different SPDK version. An `opts_size` 1722parameter is added into `spdk_app_opts_init` function. 1723 1724### event 1725 1726The `pci_whitelist` and `pci_blacklist` members of struct `spdk_app_opts` have been 1727deprecated. The new members are named `pci_allowed` and `pci_blocked` respectively. 1728 1729The `--pci-blacklist` command line option has been deprecated, replaced with 1730`--pci-blocked`. 1731 1732The `--pci-whitelist/-W` command line options have been deprecated, replaced with 1733`--pci-allowed/-A`. 1734 1735Added new experimental `dynamic` scheduler that rebalances idle threads, adjusts CPU frequency 1736using dpdk_governor and turns idle reactor cores to interrupt mode. Please see 1737[scheduler documentation](https://www.spdk.io/doc/scheduler.html) for details. 1738 1739## ioat 1740 1741The PCI BDF whitelist option has been removed from the `ioat_scan_accel_engine` RPC. 1742IOAT PCI functions can still be allowed or denied using SPDK application command 1743line options. 1744 1745### isa-l 1746 1747Updated ISA-L submodule to v2.30.0. 1748 1749### json 1750 1751A new API `spdk_jsonrpc_send_bool_response` was added to allow sending response for 1752writing json bool results into one function. 1753Update API `bdev_nvme_set_options` and add a `keep_alive_timeout_ms` parameter. Now you 1754can specify the `keep_alive_timeout` before creating NVMe bdev. 1755 1756### nbd 1757 1758Change the return type of function `spdk_nbd_stop` from void to int. And update the 1759`spdk_nbd_fini` with two parameters to make its behavior from sync to async. 1760 1761### nvme 1762 1763Directives support was added to the NVMe driver. 1764 1765Two async APIs `spdk_nvme_ctrlr_cmd_directive_receive` and `spdk_nvme_ctrlr_cmd_directive_send` 1766are added for Directive Send and Directive Receive command, respectively. 1767 1768Added a new function `spdk_nvme_ctrlr_reset_subsystem` to perform a NVMe 1769subsystem reset. Note: The NVMe-oF target does not support the subsystem reset yet. 1770 1771Add a new function `spdk_nvme_bytes_to_numd` to transfer bytes to number of 1772dwords. 1773 1774Added a new custom transport `SPDK_NVME_TRANSPORT_VFIOUSER` to enable NVMe 1775driver running with NVMe over vfio-user target. 1776 1777Added the vfio-user custom transport implementation in NVMe driver which can connect 1778to NVMe over vfio-user target via vfio-user transport. 1779 1780Added a new function `spdk_nvme_ctrlr_set_remove_cb` to remove or override 1781`remove_cb` and `remove_ctx` specified when the controller was probed. 1782 1783Added support for ZNS zone append command with new API `spdk_nvme_zns_zone_append` and 1784`spdk_nvme_zns_zone_append_with_md`. 1785 1786### nvmf 1787 1788`nvmf_fc_lld_fini` now takes callback and hence updating FC Broadcom LLD driver 1789to the latest is required. 1790 1791FC transport supported primary tagging and VMID. 1792 1793Broadcom FC LLD driver and SPDK NVMe-oF FC transport consolidated one LLD API, 1794`nvmf_fc_init_poller_queues` into another LLD API `nvmf_fc_init_q`. 1795Hence updating Broadcom FC LLD driver to the latest is required. 1796 1797The functions `destroy` and `qpair_fini` in the transport interface now accept a 1798`cb_fn` and `cb_arg` to call upon completion, and their execution can be asynchronous. 1799 1800The SPDK NVMe-oF target now supports async event notification for discovery log changes. 1801This allows the initiator to create persistent connection to discovery controller and 1802be notified of any discovery log changes. 1803 1804An `opts_size` element was added in the `spdk_nvmf_transport_opts` structure 1805to solve the ABI compatibility issue between different SPDK version. And also add 1806`opts_size` parameter in `spdk_nvmf_transport_opts_init` function. 1807 1808Added a new custom vfio-user transport implementation in NVMe-oF which can provide 1809emulated NVMe devices to QEMU and SPDK NVMe driver. 1810 1811Added new API `spdk_nvmf_tgt_listen_ext` that uses `spdk_nvmf_listen_opts` to allow 1812passing generic options to specific transport. This functionality replaces one provided 1813with `nvmf_subsystem_set_options`. Also removed `nvmf_subsystem_set_options` RPC 1814that allowed transport specific options within subsystem. 1815 1816The `trsvcid` in `nvmf_subsystem_add_listener` RPC is now optional. 1817 1818Pausing a subsystem now only pauses all admin queues. This allows for the 1819management of hosts and listeners, plus the addition of namespaces without a 1820full subsystem pause. Additionally, the target now allows for pausing 1821individual namespaces within a subsystem. To remove a namespace from a 1822subsystem, only the specific namespace must be paused. I/O will continue to 1823other namespaces while these operations execute. 1824 1825### rpc 1826 1827An new optional parameter `wait` was added to the RPC `iscsi_create_portal_group`, 1828and an new RPC `iscsi_start_portal_group` was added. They will be used not to start 1829listening on portals for a portal group until all associated target nodes are created 1830at startup, otherwise some iSCSI initiators may fail to re-login when SPDK iSCSI 1831target application restarts. 1832 1833Two optional parameter `--small-buf-pool-size` and `--large-buf-pool-size` were added 1834into `bdev_set_options` function. 1835 1836### vhost 1837 1838Added optional `packed_ring_recovery` parameter to `vhost_create_blk_controller` RPC 1839enabling packed ring live recovery. 1840 1841## v20.10 1842 1843### accel 1844 1845New APIs `spdk_accel_batch_cancel` and `spdk_idxd_batch_cancel` were added to allow 1846cancelling a batch sequence. 1847 1848### bdev 1849 1850A new `spdk_bdev_part_base_construct_ext` function has been added and the 1851`spdk_bdev_part_base_construct` has been deprecated. The 1852`spdk_bdev_part_base_construct_ext` function takes bdev name as an argument instead 1853of bdev structure to avoid a race condition that can happen when the bdev is being 1854removed between a call to get its structure based on a name and actually opening it. 1855 1856Removed `spdk_bdev_config_text` function for bdev modules to report legacy config. 1857 1858### blobstore 1859 1860A new `spdk_bdev_create_bs_dev_ext` function has been added and `spdk_bdev_create_bs_dev_from_desc` 1861function has been deprecated. 1862 1863### dpdk 1864 1865Updated DPDK submodule to DPDK 20.08. 1866 1867### env 1868 1869A new API `spdk_mem_get_fd_and_offset` was added to get file descriptor and offset for 1870a given virtual address. 1871 1872### event 1873 1874Removed `spdk_subsystem_config` callback for submodules as part of legacy config removal. 1875 1876Removed `spdk_app_get_running_config` function that printed configuration in legacy format, 1877and removed `usr1_handler` from `struct spdk_app_opts` callback that was used to call it. 1878 1879Added SPDK thread scheduler framework which is used to rebalance load among CPU cores. 1880The scheduler implementation is pluggable and two schedulers are added first, `static` scheduler 1881and `gscheduler` scheduler. 1882 1883`static` scheduler is used by default and does not reschedule threads and keeps the initial 1884assignments. 1885 1886CPU frequency governor framework and implementation are also added. 1887`dpdk_governor` is the first implementation and is based on rte_power library of DPDK. 1888`gscheduler` scheduler utilizes the CPU frequency governor. 1889 1890Scheduler and governor frameworks are experimental features. 1891 1892### fio 1893 1894Bdev fio_plugin no longer supports legacy configuration files. Options `spdk_conf` and 1895`spdk_json_conf` have to point to valid JSON configuration. 1896 1897### gpt 1898 1899Removed option to disable GPT probing via legacy configuration. Please use generic 1900bdev functionality to replicate the behavior. See `bdev_set_options --disable-auto-examine` 1901and `bdev_examine` RPC. 1902 1903### intel-ipsec-mb 1904 1905Updated intel-ipsec-mb submodule to v0.54 1906 1907### isa-l 1908 1909Updated ISA-L submodule to v2.29.0. 1910 1911### iscsi 1912 1913The iSCSI target supports iSCSI login redirection feature to achieve fault tolerance 1914and load balancing particularly in scale out iSCSI target system, which runs multiple 1915SPDK iSCSI targets. 1916 1917### json 1918 1919A new API `spdk_json_decode_object_relaxed` was added to allow parsing json values 1920even if decoder for a given key is not found. 1921 1922A new API `spdk_json_free_object` was added to free memory allocated by `spdk_json_decode_object`. 1923 1924### log 1925 1926A log flag structure, `spdk_log_register_flag`, `spdk_log_get_first_flag`, 1927`spdk_log_get_next_flag` and macro `SPDK_LOG_REGISTER_COMPONENT`, have been added which 1928enable registering and interacting with log flags. Added `SPDK_INFOLOG`, `SPDK_DEBUGLOG` 1929and `SPDK_LOGDUMP` macros to display logs for specified log flag. 1930 1931The log flags in the event framework `-L` and several SPDK applications has been enabled on 1932release builds of SPDK. On debug builds this option will additionally set 1933log print level to `SPDK_LOG_DEBUG`. 1934 1935A new API `spdk_log_enable_timestamps` and new RPC `log_enable_timestamps` was added to 1936toggle log timestamps. 1937 1938### miscellaneous 1939 1940The contents of the log_rpc library have been moved to the event library. The log_rpc 1941library now no longer exists. 1942 1943The contents of the app_rpc library have been moved to the event library. The app_rpc 1944library now no longer exists. 1945 1946The contents of the bdev_rpc library have been moved to the bdev library. The app_rpc 1947library now no longer exists. 1948 1949The bdevperf application now disables the zcopy API by default. Prior to this change, 1950bdevperf enabled using the zcopy API by default which caused a performance impact of 1951up to 25% on bdevs that don't natively support zcopy because the API emulates zero-copy 1952by allocating a buffer. The bdevperf `-x` param was renamed to `-Z` and the default 1953value changed to false. For bdevs that support zcopy, use the -Z flag to enable 1954using zcopy API. 1955 1956### net 1957 1958Net framework intended for use by user-space TCP stacks was removed. 1959 1960### nvme 1961 1962Zoned Namespace Command Set support has been added to NVMe driver, see `nvme_zns.h` for details. 1963 1964New APIs, `spdk_nvme_detach_async` and `spdk_nvme_detach_poll_async`, have been added to 1965detach multiple controllers in parallel to mitigate lengthy shutdown notification of 1966a few NVMe SSDs. 1967 1968New APIs, `spdk_nvme_ns_get_ana_group_id` and `spdk_nvme_ns_get_ana_state`, have been 1969added to get ANA group ID and ANA state of the given namespace, respectively. 1970 1971If no specific command set is requested (by setting the command_set member in the 1972`spdk_nvme_ctrlr_opts` structure), SPDK will automatically select the most appropriate 1973command set based on what the controller supports. 1974 1975### nvmf 1976 1977A new API `spdk_nvmf_subsystem_add_ns_ext` has been added and the API `spdk_nvmf_subsystem_add_ns` 1978has been deprecated. 1979 1980The NVMe-oF target now supports Asymmetric Namespace Access (ANA) Reporting to provide 1981multipath to NVMe-oF initiator. 1982 1983Add `no_wr_batching` parameter to `spdk_nvmf_transport_opts` struct to disable 1984Work Requests batching in RDMA transport. 1985 1986NVMf Target transports can now parse any additional JSON params in the nvmf_create_transport RPC 1987through the JSON context provided via spdk_nvmf_target_opts->transport_specific. 1988 1989A new RPC `nvmf_subsystem_set_options` was added to allow transport specific options within subsystem. 1990 1991A new API `spdk_nvmf_transport_stop_listen_async` was added to stop accepting new connections 1992at the provided address. `nvmf_subsystem_remove_listener` RPC now uses this function. 1993 1994A new API `spdk_nvmf_subsystem_disconnect_host` was added to disconnect all connections 1995originating from the provided hostnqn. `nvmf_subsystem_remove_host` RPC now uses this function. 1996 1997### ocf 1998 1999Updated OCF submodule to v20.03.1 2000 2001### rpc 2002 2003New optional parameters, `enable_placement_id` and `enable_quickack` were added to the 2004`sock_impl_set_options` RPC. 2005 2006A new RPC `bdev_examine_bdev` was added to allow users to examine a bdev explicitly. 2007It can be used only if bdev_auto_examine is set to false by the RPC `bdev_set_options`. 2008 2009New optional parameters `no_wr_batching` and `control_msg_num` were added to the RPC 2010'nvmf_create_transport'. 2011 2012New RPCs, `iscsi_target_node_set_redirect` and `iscsi_target_node_request_logout`, have 2013been added, and a new optional parameter `private` has been added to the RPC 2014`iscsi_create_portal_group` to support iSCSI login redirection feature. 2015 2016A new optional parameter `ana_reporting` was added to the RPC `nvmf_create_subsystem`, and 2017a new RPC `nvmf_subsystem_listener_set_ana_state` was added for ANA reporting. 2018 2019New RPCs, `nvmf_subsystem_get_listeners` and `nvmf_subsystem_get_qpairs`, were added to 2020retrieve configuration of the NVMe-oF subsystem. 2021 2022### scsi 2023 2024Two new APIs have been added `spdk_scsi_dev_construct_ext` and 2025`spdk_scsi_dev_add_lun_ext` that allow the upper layer(e.g. vhost-scsi) to 2026receive the notification when the scsi bdev has been resized. 2027 2028The `spdk_scsi_dev_construct` and `spdk_scsi_dev_add_lun` eventually may be 2029deprecated and removed. 2030 2031### sock 2032 2033The `enable_placement_id` field was added in the `struct spdk_sock_impl_opts` to 2034make the placement_id feature configurable by users. The default setting is 2035not enabled. 2036 2037The `enable_quick_ack` field was added in the `struct spdk_sock_impl_opts` to enable 2038or disable quick ack for the POSIX sock module. The default setting is not enabled. 2039 2040### thread 2041 2042`fd_group` is applied to support interrupt mode. 2043New APIs were added to support an experimental interrupt mode. This allows modules or 2044libraries to selectively register file descriptors that the spdk_thread can wait on, 2045as an alternative to polling. In v20.10, this functionality is enabled in a very small 2046subset of SPDK libraries and modules. 2047 2048### util 2049 2050A new utility named `fd_group` was add. It is now 2051implemented by epoll on Linux platform. It can be used by 2052spdk_thread and reactor to implement interrupt mode. 2053 2054Added `spdk_bit_pool` utility, see `bit_pool.h` for details. 2055 2056### vhost 2057 2058SPDK has switched to DPDK's rte_vhost library since 19.07 release, removed the internal 2059rte_vhost library which is used for DPDK older than 19.05, removed the experimental vhost 2060nvme target which depends on the internal rte_vhost library. 2061 2062### vpp 2063 2064Removed socket abstraction for VPP. 2065 2066## v20.07: SPDK CSI driver, new accel_fw commands, I/O abort support 2067 2068spdk_json_free_object() was added to free memory allocated by spdk_json_decode_object(). 2069 2070### accel 2071 2072A new API was added `spdk_accel_get_capabilities` that allows applications to 2073query the capabilities of the currently enabled accel engine back-end. 2074 2075A new capability, CRC-32C, was added via `spdk_accel_submit_crc32c`. 2076 2077The software accel engine implementation has added support for CRC-32C. 2078 2079A new capability, compare, was added via `spdk_accel_submit_compare`. 2080 2081The software accel engine implementation has added support for compare. 2082 2083Several APIs were added to `accel_engine.h` to support batched submission 2084of operations. 2085 2086Several APIs were added to `accel_engine.h` to support dualcast operations. 2087 2088### accel_fw 2089 2090The accel_fw was updated to no longer require the app to allocate an 2091accel_task on its behalf. All public APIs now take a callback arg as 2092the parameter that used to be the accel_task. 2093 2094The accel_fw API `spdk_accel_task_size` was removed as it is no longer 2095required. 2096 2097The accel_fw was updated to support compare, dualcast, crc32c. 2098 2099The accel_fw introduced batching support for all commands in all plug-ins. 2100See docs for detailed information. 2101 2102### bdev 2103 2104A new API `spdk_bdev_abort` has been added to submit abort requests to abort all I/Os 2105whose callback context match to the bdev on the given channel. 2106 2107### build 2108 2109The fio plugins now compile to `build/fio` and are named `spdk_bdev` and `spdk_nvme`. 2110Existing fio configuration files will need to be updated. 2111 2112### dpdk 2113 2114Updated DPDK submodule to DPDK 20.05. 2115 2116### env 2117 2118Several new APIs have been added to provide greater flexibility in registering and 2119accessing polled mode PCI drivers. See `env.h` for more details. 2120 2121### idxd 2122 2123The idxd library and plug-in module for the accel_fw were updated to support 2124all accel_fw commands as well as batching. Batching is supported both 2125through the library and the plug-in module. 2126 2127IDXD engine support for CRC-32C has been added. 2128 2129### ioat 2130 2131A new API `spdk_ioat_get_max_descriptors` was added. 2132 2133### nvme 2134 2135An `opts_size`element was added in the `spdk_nvme_ctrlr_opts` structure 2136to solve the ABI compatibility issue between different SPDK version. 2137 2138A new API `spdk_nvme_ctrlr_cmd_abort_ext` has been added to abort previously submitted 2139commands whose callback argument match. 2140 2141Convenience functions, `spdk_nvme_print_command` and `spdk_nvme-print_completion` were added 2142to the public API. 2143 2144A new function, `spdk_nvmf_cuse_update_namespaces`, updates the cuse representation of an NVMe 2145controller. 2146 2147A new function `qpair_iterate_requests` has been added to the nvme transport interface. ALl 2148implementations of the transport interface will have to implement that function. 2149 2150### nvmf 2151 2152The NVMe-oF target no longer supports connecting scheduling configuration and instead 2153always uses what was previously called "transport" scheduling. 2154 2155`spdk_nvmf_tgt_accept` no longer exists. The accept process now occurs automatically after 2156the creation of an nvmf target and queue pairs are assigned to poll groups by the underlying 2157transport. Further, `spdk_nvmf_transport_ops` has changed such that the accept function 2158pointer no longer takes a function pointer as an argument. Instead, transports should call 2159`spdk_nvmf_tgt_new_qpair` whenever they previously would have called that callback. 2160 2161The NVMe-oF target now supports aborting any submitted NVM or Admin command. Previously, 2162the NVMe-oF target could abort only Asynchronous Event Request commands. 2163 2164### rdma 2165 2166A new `rdma` library has been added. It is an abstraction layer over different RDMA providers. 2167Two providers are available - verbs (used by default when RDMA is enabled or enabled explicitly 2168using --with-rdma=verbs) and mlx5 Direct Verbs aka DV (enabled by --with-rdma=mlx5_dv). 2169Using mlx5_dv requires libmlx5 installed on the system. 2170 2171### rpc 2172 2173Parameter `-p` or `--max-qpairs-per-ctrlr` of `nvmf_create_transport` RPC command accepted by the 2174rpc.py script is deprecated, new parameter `-m` or `--max-io-qpairs-per-ctrlr` was added. 2175 2176Added `sock_impl_get_options` and `sock_impl_set_options` RPC methods. 2177 2178Command line parameters `-r` and `--rpc-socket` will longer accept TCP ports. RPC server 2179must now be started on a Unix domain socket. Exposing RPC on the network, as well as providing 2180proper authentication (if needed) is now a responsibility of the user. 2181 2182The `bdev_set_options` RPC has a new option, `bdev_auto_examine` to control the auto examine function 2183of bdev modules. 2184 2185New RPCs `sock_impl_get_options` and `sock_impl_set_options` been added to expose new socket features. 2186See `sock` section for more details. 2187 2188### sock 2189 2190Added `spdk_sock_impl_get_opts` and `spdk_sock_impl_set_opts` functions to set/get socket layer configuration 2191options. Options can be set independently for each implementation. 2192 2193Added `recv_buf_size` and 'send_buf_size' socket layer options. They are used only in posix implementation. 2194 2195Added `uring` based socket implementation, the code is located in module/sock/uring. This feature is only 2196available in Linux which requires kernel version is greater than 5.4.3. Currently, our CI pool added the uring 2197based socket tests for iSCSI target and also the tests for SPDK NVMe-oF tcp transport. 2198 2199Added `enable_recv_pipe` socket layer option to allow disabling of double buffering on receive. 2200New option is used only in posix implementation. 2201 2202Added `enable_zerocopy_send` socket layer option to allow disabling of zero copy flow on send. 2203New option is used only in posix implementation. 2204 2205### util 2206 2207Some previously exposed CRC32 functions have been removed from the public API - 2208`spdk_crc32_update`, `spdk_crc32_table_init`, and the `spdk_crc32_table` struct. 2209 2210### vhost 2211 2212The function `spdk_vhost_blk_get_dev` has been removed. 2213 2214## v20.04: SPDK Top, IDXD, NVMe qpair groups 2215 2216IDXD engine support for compare has been added. 2217 2218### configuration 2219 2220Legacy INI style configuration for SPDK applications has been deprecated and will be 2221removed in future release. 2222Please switch to JSON-RPC configuration files and/or RPC driven run-time configuration. 2223 2224### copy 2225 2226The copy engine library, modules and public APIs have been renamed. Use of the word `copy` 2227has been replaced with the word `accel` short for accelerator in preparation for adding new 2228capabilities in the future. Additionally, APIs for what was previously called the `memcpy` 2229engine have been renamed to identify the engine as a software accelerator. 2230 2231### crypto 2232 2233Support for AES_XTS was added for the QAT polled mode driver (pmd). The create RPC 2234`bdev_crypto_create` has 2 new optional parameters: cipher and key2. Cipher can be either 2235AES_CBC (default) or AES_XTS. AES_XTS is only valid when using the QAT polled mode driver. 2236The key2 parameter is the second key required for AES_XTS. 2237 2238### event 2239 2240Reactors now accumulate CPU stats and they are retrieved by the RPC `framework_get_reactors`. 2241 2242### idxd 2243 2244IDXD support was added in the form of a low level library that can directly 2245utilize the IDXD offload engine as well as an accel plug-in module allowing for use 2246with the generic accel framework API. IDXD is the first in a family of offload 2247engines that share the same interface, specifically DSA is added here. More info 2248can be found here: https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator 2249 2250Much of the implementation models IOAT, however the low level interfaces are very 2251different. The RPC to enable IDXD requires a configuration number as well. The 2252code includes two pre-defined configurations of IDXD groups/work queues/engines. A future 2253version will provide an interface to allow for setting of individual configuration 2254parameters. 2255 2256IDXD is not yet available so this feature should be considered experimental. It will 2257be built up with additional documentation as an ongoing activity. 2258 2259### iscsi 2260 2261The iSCSI target now creates a lightweight thread per poll group instead of assuming a pool 2262of lightweight threads already exist at start up time. A poll group is a collection of 2263unrelated iSCSI connections. Each poll group is only accessed from the associated 2264lightweight thread. 2265 2266### ftl 2267 2268Several changes have been made to the `spdk_ftl_conf`, `spdk_ftl_dev_init_ops`, and 2269`spdk_ftl_attrs` structs. Please see `include/spdk/ftl.h` for more details. 2270 2271### miscellaneous 2272 2273The `--json-ignore-init-errors` command line parameter has been added to ignore 2274initialization errors on JSON config load. 2275 2276The public header file io_channel.h has been removed. Please use thread.h which has the 2277exact same API. 2278 2279### nvme 2280 2281Exported internal nvme_ctrlr_cmd_security_receive/send() APIs as public APIs with "the spdk_" 2282prefix. 2283 2284Added `priority` field in `spdk_nvme_transport_id`, this field is used to specify the priority 2285of the NVMe-oF connection, and currently it is used for NVMe-oF tcp connection. 2286 2287A new poll group API has been added to allow for pooling of nvme qpairs across a single 2288entity which can be polled for completions. This new API consists of the `spdk_nvme_poll_group` 2289family of functions. As a result of this new API, all NVMe transports are expected to implement 2290several poll group related functions. 2291 2292A new flag, `create_only`, has been added to the `spdk_nvme_io_qpair_opts` structure. This flag 2293allows a user to call `spdk_nvme_ctrlr_get_default_io_qpair` without also connecting the qpair 2294within the context of that call. 2295 2296As a result of the `create_only` flag, two new API functions, `spdk_nvme_ctrlr_connect_io_qpair` 2297and `spdk_nvme_ctrlr_disconnect_io_qpair`, have been added to facilitate connecting newly created 2298qpairs (for example, after they have been added to a poll group) and disconnecting qpairs without 2299destroying them (for example to disconnect a qpair before migrating it to a new poll group and 2300reconnecting it). 2301 2302The functions `spdk_nvme_ctrlr_alloc_cmb_io_buffer` and `spdk_nvme_ctrlr_free_cmb_io_buffer` 2303have been changed to `spdk_nvme_ctrlr_map_cmb` and `spdk_nvme_ctrlr_unmap_cmb` respectively. 2304 2305An additional function, `spdk_nvme_ctrlr_reserve_cmb`, has been added to facilitate reserving 2306the entire size of the controller memory buffer for data transfer. 2307 2308### nvme_cuse 2309 2310`spdk_nvme_cuse_get_ctrlr_name` now takes two additional parameters, `char *name` which 2311stores the pointer to the controller name, and `size_t *size` which stores the length of 2312the name. The return type has also been changed from char * to int. 2313 2314`spdk_nvme_cuse_get_ns_name` now takes two additional parameters, `char *name` which 2315stores the pointer to the namespace name, and `size_t *size` which stores the length of 2316the name. The return type has also been changed from char * to int. 2317 2318### nvme_opal 2319 2320Several public OPAL structure definitions have been changed since the last release. 2321 2322### nvmf 2323 2324`spdk_nvmf_poll_group_destroy()` is now asynchronous and accepts a completion callback. 2325 2326The NVMe-oF target now creates a lightweight thread per poll group instead of assuming a pool 2327of lightweight threads already exist at start up time. A poll group is a collection of 2328unrelated NVMe-oF connections. Each poll group is only accessed from the associated 2329lightweight thread. 2330 2331A new struct, `spdk_nvmf_subsystem_listener`, has been added to encapsulate the subsystem specific 2332nature of a listener object. 2333 2334`spdk_nvmf_tgt_listen` no longer accepts a callback function or argument. It also returns an 2335int to indicate the status of the listen call. 2336 2337The execution of `spdk_nvme_poll_group_destroy` is now asynchronous and the function accepts 2338a cb_fn and cb_arg to call upon completion. 2339 2340The execution of `spdk_nvmf_subsystem_add_listener` is now asynchronous and the function accepts 2341a cb_fn and cb_arg to call upon completion. 2342 2343The `nvmf_transport.h` header has been made public to allow custom NVMe-oF transports to integrate 2344with NVMe-oF libraries without using internal APIs. 2345 2346### ocf 2347 2348Updated the OCF submodule to OCF v20.03 2349 2350New version of OCF comes with API changes and bug fixes 2351 2352### rpc 2353 2354A new RPC `thread_set_cpumask` has been added to set the cpumask of the thread 2355to the specified value. 2356 2357A new RPC `thread_get_pollers` has been added to retrieve pollers from SPDK threads. 2358 2359A new RPC `thread_get_io_channels` has been added to retrieve I/O channels from SPDK threads. 2360 2361A new RPC `bdev_rbd_resize` has been added to resize Ceph RBD bdevs. 2362 2363### sock 2364 2365The `spdk_sock_set_priority` function has been removed since the feature to set the sock priority 2366will be contained in two new functions, i.e., `spdk_sock_listen_ext` and `spdk_sock_connect_ext`. 2367Users may now specify the priority of the socket in the opts that they want to use. 2368 2369### spdk_top 2370 2371A new application, `spdk_top`, has been added which allows users to monitor resource consumption 2372by a running SPDK application. More information on this application can be found in 2373`app/spdk_top/README`. 2374 2375### thread 2376 2377A new function `spdk_thread_lib_init_ext` has been added, and the function 2378`spdk_thread_lib_init` has been deprecated. The user of `spdk_thread_lib_init_ext` is 2379expected to implement both functions `spdk_thread_op_fn` and `spdk_thread_op_supported_fn`. 2380`spdk_thread_op_supported_fn` is called to check whether the SPDK thread operation 2381is supported. `spdk_thread_op_fn` is called to execute the SPDK thread operation. 2382Current SPDK operation types are `SPDK_THREAD_OP_NEW` and `SPDK_THREAD_OP_RESCHED`. 2383The operation `SPDK_THREAD_OP_NEW` is called each time a new thread is created. 2384The operation `SPDK_THREAD_OP_RESCHED` is called when SPDK thread needs to be rescheduled. 2385 2386A unique ID has been added for each created SPDK thread, it is retrieved by a new function 2387`spdk_thread_get_id`, and the SPDK thread which has the specific ID is got by 2388a new function `spdk_thread_get_by_id`. 2389 2390A new function `spdk_thread_cpumask` has been added to set the current thread's cpumask 2391to the specified value. The function requires the operation `SPDK_THREAD_OP_RESCHED` 2392is supported. 2393 2394A new function `spdk_poller_register_named` has been added to set arbitrary name to the 2395created poller. If NULL, the name is set to the pointer of the poller function. 2396 2397The function `spdk_thread_poll` now measures run time per call correctly on multiple SPDK 2398threads configuration, and a new function `spdk_thread_get_last_tsc` has been added to use together. 2399 2400Voluntary termination of SPDK thread has been supported by refining the functions `spdk_thread_exit` 2401and `spdk_thread_poll`. 2402 2403### util 2404 2405New functions `spdk_sn32_lt` and `spdk_sn32_gt` have been added. They compare two sequence 2406numbers based on serial number arithmetic. 2407 2408### vhost 2409 2410Poll groups per session have been replaced by SPDK threads per vhost controller. 2411 2412### vmd 2413 2414A new function, `spdk_vmd_fini`, has been added. It releases all resources acquired by the VMD 2415library through the `spdk_vmd_init` call. 2416 2417## v20.01: Optimized thin provisioning, FTL bdev, VMD hot plug, FUSED support 2418 2419### bdev 2420 2421A new function, `spdk_bdev_set_timeout`, has been added to set per descriptor I/O timeouts. 2422 2423A new class of functions `spdk_bdev_compare*`, have been added to allow native bdev support 2424of block comparisons and compare-and-write. 2425 2426A new class of bdev events, `SPDK_BDEV_EVENT_MEDIA_MANAGEMENT`, has been added to allow bdevs 2427which expose raw media to alert all I/O channels of pending media management events. 2428 2429A new API was added `spdk_bdev_io_get_aux_buf` allowing the caller to request 2430an auxiliary buffer for its own private use. The API is used in the same manner that 2431`spdk_bdev_io_get_buf` is used and the length of the buffer is always the same as the 2432bdev_io primary buffer. 'spdk_bdev_io_put_aux_buf' frees the allocated auxiliary 2433buffer. 2434 2435### blobfs 2436 2437Added boolean return value for function spdk_fs_set_cache_size to indicate its operation result. 2438 2439Added `blobfs_set_cache_size` RPC method to set cache size for blobstore filesystem. 2440 2441### blobstore 2442 2443Added new `use_extent_table` option to `spdk_blob_opts` for creating blobs with Extent Table descriptor. 2444Using this metadata format, dramatically decreases number of writes required to persist each cluster allocation 2445for thin provisioned blobs. Extent Table descriptor is enabled by default. 2446See the [Blobstore Programmer's Guide](https://spdk.io/doc/blob.html#blob_pg_cluster_layout) for more details. 2447 2448### dpdk 2449 2450Updated DPDK submodule to DPDK 19.11. 2451 2452### env_dpdk 2453 2454`spdk_env_dpdk_post_init` now takes a boolean, `legacy_mem`, as an argument. 2455 2456A new function, `spdk_env_dpdk_dump_mem_stats`, prints information about the memory consumed by DPDK to a file specified by 2457the user. A new utility, `scripts/dpdk_mem_info.py`, wraps this function and prints the output in an easy to read way. 2458 2459### event 2460 2461The functions `spdk_reactor_enable_framework_monitor_context_switch()` and 2462`spdk_reactor_framework_monitor_context_switch_enabled()` have been changed to 2463`spdk_framework_enable_context_switch_monitor()` and 2464`spdk_framework_context_switch_monitor_enabled()`, respectively. 2465 2466### ftl 2467 2468All NVMe dependencies were removed from ftl library. 2469From now ftl library is using bdev_zone API. 2470bdev_ftl becomes virtual bdev. 2471 2472`ctrlr` and `trid` fields in `spdk_ftl_dev_init_opts` structure were replaced by 2473`base_bdev_desc`. 2474 2475`bdev_ftl_create` RPC method `trtype` and `traddr` parameters were replaced by `base_bdev` 2476parameter. 2477 2478`spdk_ftl_module_init` and `spdk_ftl_module_fini` functions were removed. 2479 2480`spdk_ftl_punit_range` and `ftl_module_init_opts` structures were removed. 2481 2482### isa-l 2483 2484Updated ISA-L submodule to commit f3993f5c0b6911 which includes implementation and 2485optimization for aarch64. 2486 2487Enabled ISA-L on aarch64 by default in addition to x86. 2488 2489### nvme 2490 2491`delayed_pcie_doorbell` parameter in `spdk_nvme_io_qpair_opts` was renamed to `delay_cmd_submit` 2492to allow reuse in other transports. 2493 2494Added RDMA WR batching to NVMf RDMA initiator. Send and receive WRs are chained together 2495and posted with a single call to ibv_post_send(receive) in the next call to qpair completion 2496processing function. Batching is controlled by 'delay_cmd_submit' qpair option. 2497 2498The NVMe-oF initiator now supports plugging out of tree NVMe-oF transports. In order 2499to facilitate this feature, several small API changes have been made: 2500 2501The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 2502A new function, `spdk_nvme_transport_available_by_name`, has been added. 2503A function table, `spdk_nvme_transport_ops`, and macro, `SPDK_NVME_TRANSPORT_REGISTER`, have been added which 2504enable registering out of tree transports. 2505 2506A new function, `spdk_nvme_ns_supports_compare`, allows a user to check whether a given namespace supports the compare 2507operation. 2508 2509A new family of functions, `spdk_nvme_ns_compare*`, give the user access to submitting compare commands to NVMe namespaces. 2510 2511A new function, `spdk_nvme_ctrlr_cmd_get_log_page_ext`, gives users more granular control over the command dwords sent in 2512log page requests. 2513 2514### nvmf 2515 2516Add SockPriority option in [Transport] section, this can be used for NVMe-oF target 2517on TCP transport to set sock priority for the incoming TCP connections. 2518 2519The NVMe-oF target now supports plugging out of tree NVMe-oF transports. In order 2520to facilitate this feature, several small API changes have been made: 2521 2522The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 2523`spdk_nvmf_tgt_get_transport`, `spdk_nvmf_transport_opts_init`, and `spdk_nvmf_transport_create` all have been 2524modified to take a string. 2525A function table, `spdk_nvmf_transport_ops`, and macro, `SPDK_NVMF_TRANSPORT_REGISTER`, have been added which 2526enable registering out of tree transports. 2527 2528Add `spdk_nvmf_tgt_stop_listen()` that can be used to stop listening for 2529incoming connections for specified target and trid. Listener is not stopped 2530implicitly upon destruction of a subsystem any more. 2531 2532A custom NVMe admin command handler has been added which allows the user to use the real drive 2533attributes from one of the target NVMe drives when reporting drive attributes to the initiator. 2534This handler can be enabled via the `nvmf_set_config` RPC. 2535Note: In a future version of SPDK, this handler will be enabled by default. 2536 2537The SPDK target and initiator both now include compare-and-write functionality with one caveat. If using the RDMA transport, 2538the target expects the initiator to send both the compare command and write command either with, or without inline data. The 2539SPDK initiator currently respects this requirement, but this note is included as a flag for other initiators attempting 2540compatibility with this version of SPDK. 2541 2542The `spdk_nvme_ctrlr_opts` struct has been extended with new field `transport_ack_timeout` which allows 2543to configure transport ACK timeout. Applicable for RDMA transport only. 2544 2545### rpc 2546 2547A new RPC, `bdev_zone_block_create`, enables creating an emulated zoned bdev on top of a standard block device. 2548 2549A new RPC, `bdev_ocssd_create`, enables creating an emulated zoned bdev on top of an Open Channel SSD. 2550 2551A new RPC, `blobfs_set_cache_size`, enables managing blobfs cache size. 2552 2553A new RPC, `env_dpdk_get_mem_stats`, has been added to facilitate reading DPDK related memory 2554consumption stats. Please see the env_dpdk section above for more details. 2555 2556A new RPC, `framework_get_reactors`, has been added to retrieve a list of all reactors. 2557 2558`bdev_ftl_create` now takes a `base_bdev` argument in lieu of `trtype`, `traddr`, and `punits`. 2559 2560`bdev_nvme_set_options` now allows users to disable I/O submission batching with the `-d` flag 2561 2562`bdev_nvme_cuse_register` now accepts a `name` parameter. 2563 2564`bdev_uring_create` now takes arguments for `bdev_name` and `block_size` 2565 2566`nvmf_set_config` now takes an argument to enable passthru of identify commands to base NVMe devices. 2567Please see the nvmf section above for more details. 2568 2569### scsi 2570 2571`spdk_scsi_lun_get_dif_ctx` now takes an additional argument of type `spdk_scsi_task`. 2572 2573### sock 2574 2575Added spdk_sock_writev_async for performing asynchronous writes to sockets. This call will 2576never return EAGAIN, instead queueing internally until the data has all been sent. This can 2577simplify many code flows that create pollers to continue attempting to flush writes 2578on sockets. 2579 2580Added `impl_name` parameter in spdk_sock_listen and spdk_sock_connect functions. Users may now 2581specify the sock layer implementation they'd prefer to use. Valid implementations are currently 2582"vpp" and "posix" and NULL, where NULL results in the previous behavior of the functions. 2583 2584### thread 2585 2586`spdk_thread_send_msg` now returns int indicating if the message was successfully 2587sent. 2588 2589A new function `spdk_thread_send_critical_msg`, has been added to support sending a single message from 2590a context that may be interrupted, e.g. a signal handler. 2591 2592Two new functions, `spdk_poller_pause`, and `spdk_poller_resume`, have been added to give greater control 2593of pollers to the application owner. 2594 2595### util 2596 2597`spdk_pipe`, a new utility for buffering data from sockets or files for parsing 2598has been added. The public API is available at `include/spdk/pipe.h`. 2599 2600### bdev 2601 2602Added spdk_bdev_io_get_nvme_fused_status function for translating bdev_io status to NVMe status 2603code for fused compare-and-write operation. 2604 2605Added spdk_bdev_get_acwu function for getting block device atomic compare and write unit size. 2606 2607## v19.10: Zoned bdev API, Opal bdev, NVMe character devices 2608 2609### rpc 2610 2611Many of SPDK's RPCs were renamed to be more consistent and intuitive in this release. 2612The old names will continue to function, but will display a deprecation warning. 2613 2614Added optional parameters '--arbitration-burst' and '--low/medium/high-priority-weight' to 2615'bdev_nvme_set_options' RPC method. 2616 2617Added optional parameter '--md-size' to 'construct_null_bdev' RPC method. 2618 2619Added optional parameters '--dif-type' and '--dif-is-head-of-md' to 'construct_null_bdev' 2620RPC method. 2621 2622Added `blobfs_detect` RPC method to detect whether a blobfs exists on given bdev. 2623 2624Added `blobfs_create` RPC method to build blobfs on given bdev. 2625 2626Added `blobfs_mount` RPC method to mount blobfs on given bdev to a host path by FUSE. 2627Then on the host path, user can directly do some file operations which will be mapped 2628to blobfs. 2629 2630### bdev 2631 2632Added new parameter `cdw0` to `spdk_bdev_io_complete_nvme_status()` and 2633`spdk_bdev_io_get_nvme_status()` that allows setting/getting 2634the NVMe completion queue DW0 entry. This allows vendor specific IO commands 2635to return command specific completion info back to the initiator. 2636 2637Added `spdk_bdev_get_write_unit_size()` function for retrieving required number 2638of logical blocks for write operation. 2639 2640New zone-related fields were added to the result of the `get_bdevs` RPC call: 2641 2642- `zoned`: indicates whether the device is zoned or a regular 2643 block device 2644- `zone_size`: number of blocks in a single zone 2645- `max_open_zones`: maximum number of open zones 2646- `optimal_open_zones`: optimal number of open zones 2647 2648The `zoned` field is a boolean and is always present, while the rest is only available for zoned 2649bdevs. 2650 2651A new `spdk_bdev_open_ext` function has been added and `spdk_bdev_open` function has been deprecated. 2652The new open function introduces requirement to provide callback function that will be called by 2653asynchronous event such as bdev removal. `spdk_bdev_open_ext` function takes bdev name as 2654an argument instead of bdev structure to avoid a race condition that can happen when the bdev 2655is being removed between a call to get its structure based on a name and actually opening it. 2656 2657New 'resize' event has been added to notify about change of block count property of block device. 2658Event is delivered only if block device was opened with `spdk_bdev_open_ext` function. 2659 2660### bdev zone 2661 2662Added new public header for zoned bdev. Zoned bdev is an extension 2663of the bdev interface. 2664 2665`spdk_bdev_get_zone_size()`, `spdk_bdev_get_max_open_zones()`, `spdk_bdev_get_optimal_open_zones()` 2666APIs were added for retrieving zoned device information. 2667`spdk_bdev_get_zone_info()` API was added for retrieving information about zones in zoned 2668device. 2669Added `spdk_bdev_zone_management()` API for changing zone state. 2670`spdk_bdev_zone_append()` and `spdk_bdev_zone_append_with_md()` APIs were added for 2671appending data to a zone. 2672Added `spdk_bdev_io_get_append location()` function for retrieving append location for I/O. 2673Added `spdk_bdev_is_zoned()` function for checking if bdev supports zoned namespace semantics. 2674 2675### bdev opal 2676 2677EXPERIMENTAL: A new opal bdev has been added to support management of 2678NVMe self-encrypting drives through the Opal specification. Users can 2679create opal bdevs from an NVMe namespace bdev, if the controller 2680containing that namespace supports Opal. Currently this is only 2681supported for namespace ID=1. The following RPCs have been added to 2682support Opal: `bdev_nvme_opal_init`, `bdev_nvme_opal_revert`, 2683`bdev_opal_create`, `bdev_opal_delete`, `bdev_opal_get_info`, 2684`bdev_opal_new_user`, `bdev_opal_set_lock_state`. 2685It does not yet support recreating the opal bdevs after application restart. 2686This bdev module should be considered very experimental, and the RPCs may 2687change significantly in future releases. 2688 2689### delay bdev 2690 2691The `bdev_delay_update_latency` has been added to allow users to update 2692a latency value for a given delay bdev. 2693 2694### compress bdev 2695 2696A new RPC `bdev_compress_get_orphans` has been added to list compress bdevs 2697that were not loaded due to a missing pm metadata file. In this state they 2698can only be deleted. 2699 2700### null bdev 2701 2702Metadata support has been added to Null bdev module. 2703 2704Protection information support has been added to Null bdev module. 2705 2706### nvme 2707 2708Added `no_shn_notification` to NVMe controller initialization options, users can enable 2709it for NVMe controllers. When the option is enabled, the controller will not do the 2710shutdown process and just disable the controller, users can start their application 2711later again to initialize the controller to the ready state. 2712 2713A controller flag `SPDK_NVME_CTRLR_WRR_SUPPORTED` was added to indicate the controller 2714can support weighted round robin arbitration feature with submission queue. 2715 2716Added `arbitration_burst` option for arbitration feature, and added three 2717`low/medium/high_priority_weight` options for weighted round robin arbitration. 2718 2719Added `spdk_nvme_ns_cmd_write_uncorrectable`. 2720 2721Added new error handling and reporting functionality. This includes several 2722new API functions to facilitate applications recovering when a qpair or 2723controller fails. 2724 2725`spdk_nvme_ctrlr_reconnect_io_qpair` - Reconnects a failed I/O qpair. 2726`spdk_nvme_ctrlr_set_trid` - Sets the trid of an existing controller. Can be used to 2727change the trid for failover cases. 2728`spdk_nvme_ctrlr_is_failed` - Returns the failed state of a controller. 2729`spdk_nvme_ctrlr_fail` - Forces a controller into a failed state. 2730 2731Modified the return behavior of several API functions to better indicate to 2732applications when a qpair is failed. This list of functions includes: 2733 2734`spdk_nvme_qpair_process_completions` 2735`spdk_nvme_ns_cmd_*` 2736`spdk_nvme_ctrlr_process_admin_completions` 2737`spdk_nvme_ctrlr_cmd_*` 2738 2739These functions now return -ENXIO when the qpair or controller on which they 2740operate is failed. 2741 2742EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux 2743kernel for controller as well as for namespace and process ioctl requests as usual 2744from linux environment. 2745 2746### nvmf 2747 2748The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts` 2749as its only parameter. This new structure contains the max_subsystems parameter previously 2750passed into that function. 2751 2752A new public API function `spdk_nvmf_get_tgt` has been added which allows users to 2753retrieve a pointer to an `spdk_nvmf_tgt` object by supplying its name. In the special 2754case where an RPC or application only creates a single target, this function can accept 2755a null name parameter and will return the only available target. 2756 2757The majority of the NVMe-oF RPCs now accept an optional tgt_name parameter. This will 2758allow those RPCs to work with applications that create more than one target. 2759 2760Three new NVMe-oF RPCs have been added `nvmf_create_target`, `nvmf_delete_target`, and 2761`nvmf_get_targets`. These new RPCs provide a basic interface for managing multiple target 2762objects. In SPDK the target object defines a unique discovery service. As of this release, 2763these RPCs are not intended to be used with the in-tree SPDK target applications, spdk_tgt and 2764nvmf_tgt, which use a single, global target structure. As such, they are not included in scripts/rpc.py 2765 2766Three new header functions have also been added to help deal with multiple targets. 2767`spdk_nvmf_tgt_get_name` takes a target pointer as an argument and returns its human readable name. 2768`spdk_nvmf_get_first_target` takes no arguments and returns the first target in the global list. 2769`spdk_nvmf_get_next_tgt` takes a target pointer as an argument and returns the next one in the global list. 2770 2771The `spdk_nvmf_tgt_accept` takes additional argument allowing to pass arbitrary context 2772information to the `new_qpair` callback. This will simplify the code when having multiple 2773nvmf targets or when retrieving the context information from globals is not suitable. 2774 2775### blobstore 2776 2777A new `spdk_bdev_create_bs_dev_from_desc` function has been added and `spdk_bdev_create_bs_dev` 2778function has been deprecated. 2779The new create function can cowork with `spdk_bdev_open_ext` function, which provides callback 2780function that will be called by asynchronous event such as bdev removal. 2781 2782### blobfs_bdev 2783 2784A new blobfs module `bdev` has been added to simplify the operations of blobfs on bdev. 2785 2786Function `spdk_blobfs_bdev_detect` is added to detect whether blobfs exists on the given block device. 2787 2788Function `spdk_blobfs_bdev_create` is added to create a blobfs on the given block device. 2789 2790Function `spdk_blobfs_bdev_mount` is added to mount a blobfs on the given block device to 2791a host path by FUSE. Then, a new thread is created dedicatedly for one mountpoint to handle 2792FUSE request by blobfs API. 2793 2794### build 2795 2796Option to build FUSE components into blobfs_bdev module for mounting a blobfs filesystem. 2797It requires the installation of libfuse3. By default, it is disabled. And it will be 2798enabled if run `./configure` with `--with-fuse` option. 2799 2800### iSCSI 2801 2802Portals may no longer be associated with a cpumask. The scheduling of 2803connections is moving to a more dynamic model. 2804 2805An new RPC `iscsi_portal_group_set_auth` has been added to set CHAP authentication 2806for discovery sessions specific for the existing iSCSI portal group. This RPC overwrites 2807the setting by the global parameters for the iSCSI portal group. 2808 2809### socket 2810 2811Added `spdk_sock_is_connected` to check whether the socket is currently connected. 2812`spdk_sock_group_poll` now returns number of events on success. 2813 2814### env 2815 2816Added `spdk_pci_device_unclaim()` function to cleanup pci claim file. 2817 2818### event 2819 2820`framework_start_init` RPC no longer stops the application on error during 2821initialization. 2822 2823### DPDK 2824 2825Updated DPDK submodule to DPDK 19.08. 2826 2827### ocf 2828 2829Updated OCF submodule to OCF v19.06 2830 2831Along with update, new cache mode 'write only' was added. 2832 2833New cache modes added to use via RPC, wi - write invalidate and wa - write around. 2834 2835New version of OCF provides fully asynchronous management API. 2836 2837## v19.07: NVMe-oF FC Transport, VMD, NVMe-oF Persistent reservations, Bdev I/O with separate metadata 2838 2839### ftl 2840 2841EXPERIMENTAL: Added ability to mirror writes to persistent write buffer cache 2842to allow for recovery from dirty shutdown event. 2843 2844Added handling of Asynchronous Nand Management Events (ANM). 2845 2846### vmd 2847 2848EXPERIMENTAL: Added Intel Volume Management Device (VMD) driver. VMD is an integrated 2849controller inside the CPU PCIe root complex. It enables virtual HBAs for the connected 2850NVMe SSDs. `spdk_vmd_init()` enumerates NVMe SSDs behind VMD device and hook them into 2851SPDK PCI subsystem. `spdk_nvme_probe()` or `spdk_nvme_connect()` can be used to connect 2852NVMe driver to the device located at the given transport ID. 2853 2854To obtain transport ID of NVMe SSD behind VMD `spdk_lspci` can be used. 2855 2856Current implementation does not support hotplug. 2857 2858### blobfs 2859 2860Blobfs file asynchronous operations were added to public APIs. 2861 2862### util 2863 2864A new file API `spdk_posix_file_load` was added to load file content into a data buffer. 2865 2866New APIs `spdk_dif_ctx_set_data_offset`, `spdk_dif_verify_stream`, 2867`spdk_dif_get_range_with_md`, `spdk_dif_get_length_with_md` have been added, 2868and existing APIs `spdk_dif_ctx_init`, `spdk_dif_set_md_interleave_iovs`, and 2869`spdk_dif_generate_stream` have been refined to insert or strip DIF into iovs 2870windows of any alignment. 2871 2872New APIs `spdk_dif_ctx_set_remapped_init_ref_tag`, `spdk_dif_remap_ref_tag`, 2873and `spdk_dix_remap_ref_tag` have been added to remap DIF reference tag. 2874 2875New APIs `spdk_dif_update_crc32c` and `spdk_dif_update_crc32c_stream` have been 2876added to compute CRC-32C checksum for extended LBA payload. 2877 2878Bdevperf and bdevio applications now support starting tests with application specific 2879RPCs. Please see helper Python scripts in their respective directories. 2880This is a move towards simpler RPC-only configuration for all main 2881and auxiliary applications. 2882 2883Legacy INI style configuration for SPDK applications will become deprecated in SPDK 19.10, 2884and removed in SPDK 20.01. Please consider moving to JSON-RPC configuration files and/or 2885RPC driven run-time configuration. 2886 2887### nvmf 2888 2889EXPERIMENTAL: A Fibre Channel transport that supports Broadcom HBAs has been 2890added. This depends on the FC HBA driver at 2891https://github.com/ecdufcdrvr/bcmufctdrvr. See [the documentation](https://spdk.io/doc/nvmf.html#nvmf_fc_transport) 2892for more information. 2893 2894Persistent reservation emulation has been added to the NVMe-oF target. Persistent reservation 2895state is stored in a JSON file on the local filesystem between target restart. To support this, 2896an optional parameter to the RPC method `nvmf_subsystem_add_ns` called `--ptpl-file` was added. 2897This allows the user to specify which file to store the persistent reservation state in. Note 2898that this is done per namespace. 2899 2900The c2h success optimization under which a command capsule response is not sent 2901for reads is turned on by default. A config knob was added to allow disabling 2902the optimization. This will mostly be used for integration testing with 5.0.x kernels 2903while some compatibility fixes make their way down the pipeline for 5.1.x kernels. 2904 2905The sock priority setting of the TCP connection owned by the tcp transport is added. It is 2906used to optimize the TCP connection performance under designated traffic classes. And the 2907priority is used to differentiate the sock priority between SPDK NVMe-oF TCP target application 2908and other TCP based applications. 2909 2910Shared receive queue can now be disabled even for NICs that support it using the 2911`nvmf_create_transport` RPC method parameter `no_srq`. The actual use of a shared 2912receive queue is predicated on hardware support when this flag is not used. 2913 2914spdk_nvmf_get_optimal_poll_group was added, which is used to return the optimal 2915poll group for the qpair. And `ConnectionScheduler` configuration is added into the 2916[Nvmf] section in etc/spdk/nvmf.conf.in to demonstrate how to configure the connection 2917scheduling strategy among different spdk threads. 2918 2919Added infrastructure to retrieve global and per poll group NVMf statistics. 2920 2921DIF strip and insert is now supported for TCP transport. When it is enabled, DIF 2922setting is not exposed to the NVMe-oF initiator, and DIF is attached into data 2923for write I/O and stripped from data for read I/O. 2924 2925Added a field `dif_insert_or_strip` to struct spdk_nvmf_transport_opts, and 2926updated the related rpc function nvmf_create_transport to make this 2927configurable parameter available to users. The `dif_insert_or_strip` is relevant 2928for TCP transport for now and used to configure the DIF strip and insert. 2929 2930Added infrastructure to retrieve NVMf transport statistics. 2931 2932### notify 2933 2934The function `spdk_notify_get_types()` and `spdk_notify_get_events()` were 2935renamed to `spdk_notify_foreach_type()` and `spdk_notify_foreach_event()`, 2936respectively. And update type name of callback accordingly. 2937 2938### bdev 2939 2940The format of the data returned by the get_bdevs_iostat RPC has changed to 2941make it easier to parse. It now returns an object with a "ticks" object 2942and "bdevs" array with the per-bdev statistics. 2943 2944A new bdev module `delay` has been added which simulates a drive latency when placed 2945on top of a Null bdev. This module is intended only for testing and can be created using 2946the new RPC `bdev_delay_create`. That RPC takes the name of the underlying bdev as well 2947as average and p99 latency arguments for both read and write operations. Average latency is 2948defined as a value close to what you would expect a perf tool such as FIO to report back as 2949the mean latency of all I/O submitted to the drive. p99 latency is defined as the value one 2950would expect the drive to see the slowest 1% of I/O report. For underlying drives with already 2951significant latency, the latency values provided to the drive will be additive. This should be 2952taken into account if trying to achieve an artificial latency on top of an nvme drive or aio device. 2953 2954DIF reference tag remapping is now supported for partition type virtual bdev 2955modules. When using partition type virtual bdevs, block address space is 2956remapped during I/O processing and DIF reference tag is remapped accordingly. 2957 2958Added spdk_bdev_*_with_md() functions allowing for IO with metadata being transferred in 2959separate buffer. To check support for separate metadata, use spdk_bdev_is_md_separate(). 2960 2961All bdevs now have a UUID. For devices whose backing hardware does not provide a UUID, 2962one is automatically generated. Across runs of SPDK, bdevs whose UUID is automatically 2963generated may change. 2964 2965A new virtual bdev module `compress` has been added to provide compression services on top of 2966a thinly provisioned logical volume. See documentation for complete details. 2967 2968### nvme 2969 2970Added an optional parameter `--io-queue-requests` to RPC `set_bdev_nvme_options`, which 2971can be used to change the number of requests allocated for one NVMe I/O queue. For 2972very big I/O size, e.g. 128MiB, with this option user will not get an error due to 2973limited requests in NVMe driver layer. 2974 2975Added spdk_nvme_ctrlr_get_transport_id() to get the transport ID from a 2976previously attached controller. 2977 2978Nvme Opal library spdk_opal_cmd deprecated. Adding separate command APIs. 2979NVMe Opal library add support for activating locking SP which will make the transaction 2980from "Manufactured-Inactive" state to "Manufactured" state. Upon successfully invoking 2981of this method, lock and unlock features will be enabled. 2982NVMe Opal library add support for locking/unlocking range and list locking range info. 2983NVMe opal library add support for multiuser. Admin can enable user and add user to specific 2984locking range and the user can lock/unlock his range. 2985 2986Added spdk_nvme_ctrlr_io_cmd_raw_no_payload_build() allowing a caller to pass 2987a completely formed command to an NVMe submission queue (buffer addresses and all). 2988This is supported on the PCIe transport only. 2989 2990Added spdk_nvme_get_ctrlr_registers() to return a pointer to the virtual address 2991of the NVMe controller registers. This is supported on the PCIe transport only. 2992 2993Added additional options to the spdk_nvme_ctrlr_alloc_qpair() option parameter 2994structure to allow caller to override the virtual and optionally physical address 2995of the submission and completion queue pair to be created. This is supported on 2996the PCIe transport only. 2997 2998Added `disable_error_logging` to struct spdk_nvme_ctrlr_opts, which disables 2999logging of failed requests. By default logging is enabled. 3000 3001Added spdk_nvme_qpair_print_command(), spdk_nvme_qpair_print_completion() and 3002spdk_nvme_cpl_get_status_string(). Allowing for easier display of error messages. 3003 3004Added support for NVMe Sanitize command. 3005 3006### env 3007 3008The parameter `free_space` has been added to spdk_ring_enqueue() to wait when 3009the ring is almost full and resume when there is enough space available in 3010the ring. 3011 3012A new API `spdk_mempool_lookup` has been added to lookup the memory pool created 3013by the primary process. 3014 3015Added spdk_pci_get_first_device() and spdk_pci_get_next_device() to allow 3016iterating over PCI devices detected by SPDK. Because of this, all SPDK APIs 3017to attach/detach PCI devices are no longer thread safe. They are now meant to 3018be called from only a single thread only, the same only that called spdk_env_init(). 3019This applies to the newly added APIs as well. 3020 3021### vpp 3022 3023SPDK now supports VPP version 19.04.2, up from VPP 18.01. 3024 3025VPP socket abstraction now uses VPP Session API, instead of VLC (VPP Communications Library). 3026This allows for better control over sessions and queues. 3027Please see VPP documentation for more details: 3028[VPP Host Stack](https://wiki.fd.io/view/VPP/HostStack) 3029 3030### sock 3031 3032Add spdk_sock_get_optimal_sock_group(), which returns the optimal sock group for 3033this socket. When a socket is created, it is often assigned to a sock group using 3034spdk_sock_group_add_sock so that a set of sockets can be polled more efficiently. 3035For some network devices, it is optimal to assign particular sockets to specific 3036sock groups. This API is intended to provide the user with that information. 3037 3038spdk_sock_group_get_ctx() was added to return the context of the spdk_sock_group. 3039spdk_sock_group_create() is updated to allow input the user provided ctx. 3040spdk_sock_set_priority() is added to set the priority of the socket. 3041 3042### rpc 3043 3044Added thread_get_stats RPC method to retrieve existing statistics. 3045 3046Added nvmf_get_stats RPC method to retrieve NVMf subsystem statistics. 3047 3048Response buffers for RPC requests are now always pre-allocated, which implies 3049that all spdk_jsonrpc_begin_result() calls always succeed and return a valid 3050buffer for JSON response. RPC calls no longer need to check if the buffer is 3051non-NULL. 3052 3053Added SPDK_RPC_REGISTER_ALIAS_DEPRECATED to help with deprecation process when 3054renaming existing RPC. First time a deprecated alias is used, it will print 3055a warning message. 3056 3057RPC `get_rpc_methods` was renamed `rpc_get_methods`. The old name is still usable, 3058but is now deprecated. 3059 3060### blobstore 3061 3062A snapshot can now be deleted if there is only a single clone on top of it. 3063 3064### build 3065 3066Preliminary support for cross compilation is now available. Targeting an older 3067CPU on the same architecture using your native compiler can be accomplished by 3068using the `--target-arch` option to `configure` as follows: 3069 3070~~~bash 3071./configure --target-arch=broadwell 3072~~~ 3073 3074Additionally, some support for cross-compiling to other architectures has been 3075added via the `--cross-prefix` argument to `configure`. To cross-compile, set CC 3076and CXX to the cross compilers, then run configure as follows: 3077 3078~~~bash 3079./configure --target-arch=aarm64 --cross-prefix=aarch64-linux-gnu 3080~~~ 3081 3082### vhost 3083 3084A security vulnerability has been identified and fixed in SPDK Vhost-SCSI target. 3085A malicious client (e.g. a virtual machine) could send a carefully prepared, 3086invalid I/O request to crash the entire SPDK process. All users of SPDK Vhost-SCSI 3087target are strongly recommended to update. All SPDK versions < 19.07 are affected. 3088 3089By default, SPDK will now rely on upstream DPDK's rte_vhost instead of its fork 3090located inside SPDK repo. The internal fork is still kept around to support older 3091DPDK versions, but is considered legacy and will be eventually removed. 3092 3093`configure` will now automatically use the upstream rte_vhost if the used DPDK 3094version is >= 19.05. 3095 3096spdk_vhost_init() is now asynchronous and accepts a completion callback. 3097 3098### iscsi target 3099 3100A security vulnerability has been identified and fixed in SPDK iSCSI target. 3101A malicious client (e.g. an iSCSI initiator) could send a carefully prepared, 3102invalid I/O request to crash the entire SPDK process. All users of SPDK iSCSI 3103target are strongly recommended to update. All SPDK versions < 19.07 are affected. 3104 3105### thread 3106 3107Exposed spdk_set_thread() in order for applications to associate 3108with SPDK thread when necessary. 3109 3110Added spdk_thread_destroy() to allow framework polling the thread to 3111release resources associated with that thread. 3112 3113## v19.04: Compression bdev, Notification library, NVMe Opal support 3114 3115### nvme 3116 3117Added asynchronous probe support. New APIs spdk_nvme_probe_async(), 3118spdk_nvme_connect_async() and spdk_nvme_probe_poll_async() were added to enable 3119this feature. spdk_nvme_probe_async() and spdk_nvme_connect_async() return a 3120context associated with the specified controllers. Users then call 3121spdk_nvme_probe_poll_async() until it returns 0, indicating that the operation 3122completed. 3123 3124A new qpair creation option, delay_pcie_doorbell, was added. This can be passed 3125to spdk_nvme_alloc_io_qpair(). This makes the I/O submission functions, such as 3126spdk_nvme_ns_writev(), skip ringing the submission queue doorbell. Instead the 3127doorbell will be rung as necessary inside spdk_nvme_qpair_process_completions(). 3128This can result in significantly fewer MMIO writes to the doorbell register 3129under heavy load, greatly improving performance. 3130 3131spdk_nvme_ctrlr_get_regs_cmbsz() was added to report the size of the controller 3132memory buffer, if available. 3133 3134spdk_nvme_ctrlr_get_flags() was added to return controller feature 3135flags. Two flags are currently tracked: 3136SPDK_NVME_CTRLR_SGL_SUPPORTED 3137SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED 3138 3139The NVMe hotplug poller is now able to detach devices hot-removed from the system 3140via `/sys/bus/pci/devices/<bdf>/remove` and `/sys/bus/pci/devices/<bdf>/driver/unbind`. 3141 3142Opal support was added for scan, take ownership, revert TPer, and dumping device 3143info. The nvme_manage tool can be used to perform these operations. The public 3144API functions are spdk_nvme_ctrlr_security_receive() and 3145spdk_nvme_ctrlr_security_send(). This module should be considered experimental 3146pending additional features and tests. 3147 3148The NVMe-oF initiator is now able to transfer IO whose size is more than 128KiB 3149if the corresponding NVMe-oF target allows. 3150 3151### raid 3152 3153Added new strip_size_kb rpc param on create to replace the more ambiguous 3154strip_size. The strip_size rpc param is deprecated. 3155 3156Changed the raid bdev product_name from "Pooled Device" to "Raid Volume" 3157 3158### thread 3159 3160Added spdk_thread_has_pollers() function to verify if there are any registered 3161pollers to be run on the thread. Added spdk_thread_is_idle() function to check 3162if there are any scheduled operations to be performed on the thread at given 3163time. 3164 3165spdk_thread_create() now takes an optional CPU affinity mask that is passed to 3166the scheduler. 3167 3168spdk_thread_lib_init() now takes an optional context size. For each thread 3169created, an additional region of memory of this size will be allocated. A 3170pointer to this region of memory can be obtained by calling 3171spdk_thread_get_ctx(). The inverse operation is also available via 3172spdk_thread_get_from_ctx(). 3173 3174spdk_thread_poll() now optionally accepts the current time, in ticks. This can 3175be used to avoid some calls to spdk_get_ticks() internally. 3176 3177spdk_app_start() now only accepts a single context argument. 3178 3179### bdev 3180 3181An new API `spdk_bdev_get_data_block_size` has been added to get size of data 3182block except for metadata. 3183 3184spdk_vbdev_register() has been deprecated. spdk_bdev_register() should be used 3185instead. 3186 3187A mechanism for acquiring and releasing data buffers from bdev modules, used 3188to perform zero copy operations, was added. 3189 3190New APIs spdk_bdev_get_md_size(), spdk_bdev_is_md_interleaved(), spdk_bdev_get_dif_type(), 3191spdk_bdev_is_dif_head_of_md(), and spdk_bdev_is_dif_check_enabled() have been 3192added to get metadata and DIF settings. 3193 3194Bdevs claimed by the `examine_config` callback will be now further examined in the 3195`examine_disk` callback. 3196 3197spdk_bdev_io_get_io_channel() was added as a convenient way to get an io_channel 3198from a bdev_io. 3199 3200### ocf 3201 3202Updated OCF submodule to OCF v19.3.2 3203 3204Added support for many-to-one configuration for OCF bdev. 3205Multiple core devices can now be cached on single cache device. 3206 3207Added persistent metadata support, allowing to restore cache state after shutdown. 3208During start of SPDK application, the devices are examined and if OCF metadata 3209is present - appropriate OCF bdevs will be recreated. 3210 3211Added Write-Back mode support. In this mode, data is first written to 3212caching device and periodically synchronized to the core devices. 3213Dirty data is saved as persistent metadata on cache device, 3214allowing for safe restore during application restart. 3215For more details please see OCF documentation: 3216[OpenCAS cache configuration](https://open-cas.github.io/cache_configuration.html) 3217 3218### NVMe-oF Target 3219 3220Support for per-device shared receive queues in the RDMA transport has been added. 3221It is enabled by default for any device that supports it. 3222 3223The size of a shared receive queue is defined by transport configuration file parameter 3224`MaxSRQDepth` and `nvmf_create_transport` RPC method parameter `max_srq_depth`. 3225Default size is 4096. 3226 3227Add model number as parameter to construct_nvmf_subsystem (-d option), 3228rather than using hardcoded define. 3229 3230DIF passthrough feature has been added. DIF setting of the allocated bdevs is 3231exposed to the NVMe-oF initiator and data with DIF from the NVMe-oF initiator is 3232passed through to the allocated bdevs. 3233 3234### env 3235 3236The `phys_addr` parameter in spdk_malloc() and spdk_zmalloc() has been deprecated. 3237For retrieving physical addresses, spdk_vtophys() should be used instead. 3238 3239spdk_realloc() has been added to reallocate DMA/shared memory. 3240 3241spdk_pci_device_is_removed() has been added to let the upper-layer SPDK drivers know 3242that device has a pending external hotremove request. 3243 3244spdk_env_fini() and spdk_env_dpdk_post_fini() were added to release any resources 3245allocated by spdk_env_init() or spdk_env_dpdk_post_init() respectively. It is expected 3246that common usage of those functions is to call them just before terminating the process. 3247 3248Added spdk_iommu_is_enabled() to report if SPDK application is using IOMMU for DMA. 3249 3250### DPDK 3251 3252Dropped support for DPDK 17.07 and earlier, which SPDK won't even compile with right now. 3253 3254Updated DPDK submodule to DPDK 19.02. 3255 3256### rpc 3257 3258New `get_spdk_version` RPC method is introduced to get version info of the running SPDK application. 3259 3260The `start_nbd_disk` RPC method now take nbd_device as an optional parameter. If nbd_device 3261is specified, use that specified nbd device. If it's not specified, pick available one. 3262 3263### iSCSI target 3264 3265DIF strip and insert is now supported. DIF settings are not exposed to the iSCSI initiator. 3266DIF is attached into data for write I/O and stripped from data for read I/O. 3267 3268### vhost 3269 3270Added experimental support for running with the external, upstream rte_vhost library. 3271This can be enabled by configuring SPDK with an `--without-internal-vhost-lib` flag. 3272The minimum supported rte_vhost version (DPDK version) is 19.05-rc1. 3273 3274As a result of fuzz testing, a lot of data races in vhost-scsi LUN hotplug path were identified and 3275fixed. Those data races could have potentially resulted in SPDK crashes, RPC hangs, or memory leaks 3276if Vhost-SCSI LUN hotplug RPCs were executed while connected VMs were in the middle of restarting. 3277 3278The SCSI target id in `add_vhost_scsi_lun` RPC is now optional. If `-1` is passed, the first 3279unoccupied target id will be used. 3280 3281### AIO 3282 3283AIO bdev module can now reap I/O completions directly from userspace, significantly improving 3284the overall performance. 3285 3286### blobfs 3287 3288Synchronous IO operations no longer use spdk_io_channel, but instead use 3289spdk_fs_thread_ctx. The behavior is otherwise identical. 3290 3291### OCF 3292 3293Added support for caching multiple bdevs using a single bdev as a cache. 3294 3295### notify 3296 3297Added the notify library that provides a high performance local event bus 3298between libraries. Example usage was added to bdev module, which reports 3299notifications for added and removed bdevs via RPC. 3300 3301### sock 3302 3303Added new API spdk_sock_readv() to the sock library for performing vectored 3304reads. 3305 3306### event 3307 3308The function spdk_subsystem_init() no longer requires spdk_event as an argument. 3309 3310Changed API of spdk_subsystem_config_json() to no longer be asynchronous. 3311 3312### io_uring 3313 3314A bdev module that issues I/O to kernel block devices using the new io_uring Linux kernel 3315API was added. This module requires liburing. 3316 3317### build 3318 3319Options to easily compile with profile guided optimization have been added to 3320`configure`. To use profile guided optimization with SPDK, run 3321`./configure --with-pgo-capture`, build SPDK, then run a workload of your 3322choosing. Then, simply run `./configure --with-pgo-enable` and recompile to 3323build using the generated profile data. Profile guided optimization can yield 3324very large performance improvements, especially on GCC 8 and clang 7. This may 3325be combined with link time optimization which has been available under the 3326`--enable-lto` configure option for several releases. 3327 3328### compression bdev/reduce library 3329 3330Added "reduce" block compression scheme based on using SSDs for storing 3331compressed blocks of storage and persistent memory for metadata. Please see 3332[compression](https://spdk.io/doc/bdev.html) for more details. 3333 3334## v19.01: NVMe-oF TCP/IP Transport, Open Channel SSD Flash Translation Layer, Caching bdev based on OCF, ISA-L Support, DIF/DIX library 3335 3336### ocf bdev 3337 3338New virtual bdev module based on [Open CAS Framework](https://open-cas.github.io/) has been added. 3339This module allows for the use of one bdev to act as a high performance cache in front of another bdev. 3340Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_cas) for more details. 3341Only write through mode is currently supported and this feature is considered experimental. 3342 3343### event framework 3344 3345For `spdk_app_parse_args`, add return value to the callback which parses application 3346specific command line parameters to protect SPDK applications from crashing by invalid 3347values from user input. 3348 3349By default, all SPDK applications will now reserve all hugepages at runtime. The pre-reserved 3350memory size can be still set with `-s` or `--mem-size` option, although the default value 3351was reduced down to 0. 3352 3353A custom hugetlbfs directory can now be specified via spdk_app_opts. 3354This can be used to configure hugepages with different sizes, a different size limit, 3355or different access permissions than the system's default hugepage pool. 3356SPDK applications can specify a custom hugetlbfs mount with the `--huge-dir` option. 3357 3358### environment 3359 3360spdk_vtophys() has been refactored to accept length of the translated region as a new 3361parameter. The function will now update that parameter with the largest possible value 3362for which the memory is contiguous in the physical memory address space. 3363 3364The following functions were removed: 3365 3366- spdk_pci_nvme_device_attach() 3367- spdk_pci_nvme_enumerate() 3368- spdk_pci_ioat_device_attach() 3369- spdk_pci_ioat_enumerate() 3370- spdk_pci_virtio_device_attach() 3371- spdk_pci_virtio_enumerate() 3372 3373They were replaced with generic spdk_pci_device_attach() and spdk_pci_enumerate() which 3374require a new spdk_pci_driver object to be provided. It can be one of the following: 3375 3376- spdk_pci_nvme_get_driver() 3377- spdk_pci_ioat_get_driver() 3378- spdk_pci_virtio_get_driver() 3379 3380spdk_pci_hook_device() and spdk_pci_unhook_device() were added. Those allow adding a virtual 3381spdk_pci_device into the SPDK PCI subsystem. A virtual device calls provided callbacks for 3382each BAR mapping request or PCI config access. It's attachable with spdk_pci_device_attach() 3383or spdk_pci_enumerate() like any other device. 3384 3385A new spdk_pause() function was added to pause CPU execution for an implementation specific 3386amount of time. Quoting from DPDK function this is based on: "This call is intended for 3387tight loops which poll a shared resource or wait for an event. A short pause within the loop 3388may reduce the power consumption." 3389 3390A new public header file env_dpdk.h has been introduced, and function spdk_env_dpdk_post_init 3391is added into it. If user is using DPDK, and already called rte_eal_init, then include 3392include/spdk/env_dpdk.h, and call spdk_env_dpdk_post_init() instead of spdk_env_init. 3393 3394ISA-L has been added as an SPDK submodule. ISA-L is enabled by default on x86 architecture 3395to accelerate algorithms such as CRC for iSCSI and NVMe-oF. Users may still disable ISA-L 3396by explicitly passing --without-isal to the configure script. 3397 3398### util 3399 3400A new uuid API `spdk_uuid_copy` was added to make a copy of the source uuid. 3401 3402An new parameter `init_crc` representing the initial CRC value was added to 3403`spdk_crc16_t10dif`. The parameter can be used to calculate a CRC value spanning 3404multiple separate buffers. 3405 3406New DIF APIs were added to generate and verify DIF by byte granularity for both DIF and DIX 3407formats. Among them, DIF with copy APIs will be usable to emulate DIF operations such as DIF 3408insert and strip. 3409 3410Added `spdk_strtol` and `spdk_strtoll` to provide additional error checking around `strtol` 3411and `strtoll`. 3412 3413Added `spdk_sprintf_append_realloc` and `spdk_vsprintf_append_realloc` for appending a string 3414with automatic buffer re-allocation. 3415 3416### nvme 3417 3418Wrapper functions spdk_nvme_ctrlr_security_send() and spdk_nvme_ctrlr_security_receive() are 3419introduced to support further security protocol development. 3420 3421admin_timeout_ms was added to NVMe controller initialization options, users 3422can change the default value when probing a controller. 3423 3424Add two new fields "header_digest" and "data_digest" in struct spdk_nvme_ctrlr_opts, 3425it will be used to enable the digest support for the NVMe/TCP transport. 3426 3427Add a new TCP/IP transport(located in lib/nvme/nvme_tcp.c) in nvme driver. With 3428this new transport, it can be used to connect the NVMe-oF target with the 3429same TCP/IP support. 3430 3431Added API, spdk_nvme_ctrlr_is_discovery(), to indicate whether the ctrlr 3432arg refers to a Discovery Controller or not. 3433 3434Added an API function `spdk_nvme_host_id_parse` and corresponding object `spdk_nvme_host_id` 3435for parsing host address and host service ID arguments on a per connection basis. 3436 3437The RPC `construct_nvme_bdev` now allows a user to specify a source address and service id for the host to 3438use when connecting to the controller backing the NVMe bdev. 3439 3440### NVMe-oF Target 3441 3442The `spdk_nvmf_tgt_opts` struct has been deprecated in favor of `spdk_nvmf_transport_opts`. 3443Users will no longer be able to specify target wide I/O parameters. `spdk_nvmf_tgt_listen` 3444will also no longer implicitly initialize a transport with the default target options (since 3445there are none). Instead, a user must manually instantiate the transport with `spdk_nvmf_transport_create` 3446prior to calling `spdk_nvmf_tgt_listen`. 3447 3448Related to the previous change, the rpc `set_nvmf_target_options` has been renamed to 3449`set_nvmf_target_max_subsystems` to indicate that this is the only target option available for the user to edit. 3450 3451Added fields `num_shared_buffers` and `buf_cache_size` in struct spdk_nvmf_transport_opts, 3452and also updated the related rpc function nvmf_create_transport, to make this 3453configurable parameter available to users. The `num_shared_buffers` is used to 3454configure the shared buffer numbers of the transport used by RDMA or TCP transport. 3455`buf_cache_size` configures number of shared buffers to cache per poll group. 3456 3457### nvmf 3458 3459Add a new TCP/IP transport (located in lib/nvmf/tcp.c). With this transport, 3460the SPDK NVMe-oF target can have a new transport, and can serve the NVMe-oF 3461protocol via TCP/IP from the host. 3462 3463Added optional mechanism to modify the RDMA transport's behavior when creating protection domains and registering memory. 3464By default, the RDMA transport will use the ibverbs library to create protection domains and register memory. 3465Using `spdk_nvme_rdma_init_hooks` will subvert that and use an existing registration. 3466 3467### bdev 3468 3469Added `enable_bdev_histogram` and `get_bdev_histogram` RPC commands to allow gathering latency data for specified bdev. 3470Please see [documentation](https://spdk.io/doc/bdev.html#rpc_bdev_histogram) for more details. 3471 3472Added `required_alignment` field to `spdk_bdev`, that specifies an alignment requirement for data buffers associated with an spdk_bdev_io. 3473Bdev layer will automatically double buffer any spdk_bdev_io that violates this alignment, before the spdk_bdev_io is submitted to the bdev module. 3474 3475On shutdown, bdev unregister now proceeds in top-down fashion, with 3476claimed bdevs skipped (these will be unregistered later, when virtual 3477bdev built on top of the respective base bdev unclaims it). This 3478allows virtual bdevs to be shut down cleanly as opposed to the 3479previous behavior that didn't differentiate between hotremove and 3480planned shutdown. 3481 3482The `delete_bdev` RPC is now deprecated. Users should instead use the specific deletion RPC 3483for the bdev type to be removed (i.e. delete_malloc_bdev). 3484 3485Added support for separate bandwidth rate limits for read and write to QoS in bdev layer. 3486 3487Bdev I/O statistics now track unmap operations. 3488 3489### logical volumes 3490 3491Logical volume bdev can now be marked as read only using `set_read_only_lvol_bdev` RPC. 3492This allows for basing clones on top of lvol_bdev without first creating a snapshot. 3493 3494Added option to change method for data erasure when deleting lvol or resizing down. 3495Default of unmapping clusters can now be changed to writing zeroes or no operation. 3496 3497Added option to change method for erasing data region on lvol store creation. 3498Default of unmapping can now be changed to writing zeroes or no operation. 3499 3500### log 3501 3502"trace flags" are now referred to as "log flags" in the SPDK log API. The 3503set_trace_flag, clear_trace_flag and get_trace_flags RPCs are now deprecated, 3504and set_log_flag, clear_log_flag and get_log_flags RPCs have been added. 3505 3506### trace 3507 3508New `trace_record` application was added. It can be used to poll spdk trace shm file and 3509append any new trace entries into another specified file. This can help retain those entries 3510that would otherwise be overwritten in the shm file. See 3511[Capturing sufficient trace events](https://spdk.io/doc/nvmf_tgt_tracepoints.html#capture_trace_events) 3512for more details. 3513 3514Number of trace entries in circular buffer per lcore can now be assigned by starting SPDK app 3515with argument "--num-trace-entries <NUM>" provided. 3516 3517New `get_tpoint_group_mask` RPC was added to get current tpoint_group_mask, and 3518each tpoint group status. 3519New `enable_tpoint_group` and `disable_tpoint_group` RPC were added to enable or 3520disable a specific tpoint group. 3521 3522### ftl 3523 3524EXPERIMENTAL: Added basic flash translation layer module allowing for using Open Channel SSDs as 3525block devices. The module is split into the library (located in lib/ftl) and bdev_ftl 3526(lib/bdev/ftl). See the [documentation](https://spdk.io/doc/ftl.html) for more details. 3527 3528### vhost 3529 3530A security vulnerability has been identified and fixed in the SPDK vhost target. A malicious 3531vhost client (i.e. virtual machine) could carefully construct a circular descriptor chain which 3532would result in a partial denial of service in the SPDK vhost target. These types of descriptor 3533chains are now properly detected by the vhost target. All SPDK vhost users serving untrusted 3534vhost clients are strongly recommended to upgrade. (Reported by Dima Stepanov and Evgeny 3535Yakovlev.) 3536 3537Vhost SCSI and Vhost Block devices can now accept multiple connections on the same socket file. 3538Each connection (internally called a vhost session) will have access to the same storage, but 3539will use different virtqueues, different features and possibly different memory. 3540 3541### vhost scsi 3542 3543SCSI target hotremove can now be performed even without the VIRTIO_SCSI_F_HOTPLUG feature negotiated. 3544Regardless of VIRTIO_SCSI_F_HOTPLUG support, the hotremoval will be still reported through SCSI sense codes. 3545 3546### DPDK 3547 3548DPDK submodule was updated to DPDK 18.11. Note that SPDK does not fully leverage the new 3549multi-process device hotplug yet and continues to work the same way it always did. 3550 3551Dropped support for DPDK 16.07 and earlier, which SPDK won't even compile with right now. 3552 3553### RPC 3554 3555The following RPC commands deprecated in the previous release are now removed: 3556 3557- construct_virtio_user_scsi_bdev 3558- construct_virtio_pci_scsi_bdev 3559- construct_virtio_user_blk_bdev 3560- construct_virtio_pci_blk_bdev 3561- remove_virtio_scsi_bdev 3562- construct_nvmf_subsystem 3563 3564### Miscellaneous 3565 3566The configure options `--with-raid` and `--without-raid` that were deprecated in the previous 3567release are now removed. 3568 3569### nbd 3570 3571Starting nbd using `spdk_nbd_start` is now performed asynchronously. 3572 3573### net framework 3574 3575Net framework initialization and finish is now done asynchronously. 3576 3577### rpc 3578 3579Added `spdk_rpc_is_method_allowed` function for checking whether method is permitted in a given state. 3580Added `spdk_rpc_get_state` to check current state of RPC server. 3581RPC `wait_subsystem_init` has been added to allow clients to block until all subsystems are initialized. 3582 3583### json rpc 3584 3585JSON RPC client is now running in non-blocking mode. Requests are sent and received during spdk_jsonrpc_client_poll. 3586JSON RPC server can now receive a callback on connection termination or server shutdown using `spdk_jsonrpc_conn_add_close_cb` 3587and `spdk_jsonrpc_conn_del_close_cb`. 3588 3589## v18.10: Dynamic memory allocation, Crypto Virtual Bdev, jsonrpc-client, SPDKCLI iSCSI and NVMe-oF support 3590 3591### nvme 3592 3593spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive() 3594were added to support sending or receiving security protocol data to or from 3595nvme controller. 3596 3597spdk_nvme_ns_get_extended_sector_size() was added. This function includes 3598the metadata size per sector (if any). spdk_nvme_ns_get_sector_size() still 3599returns only the data size per sector, not including metadata. 3600 3601New `send_nvme_cmd` RPC was added to allow sending NVMe commands directly to NVMe controller. 3602See the [send_nvme_cmd](http://spdk.io/doc/jsonrpc.html#rpc_send_nvme_cmd) documentation 3603for more details. 3604 3605### Build System 3606 3607New `configure` options, `--with-shared` and `--without-shared` 3608[default], provide the capability to build, or not, SPDK shared libraries. 3609This includes the single SPDK shared lib encompassing all of the SPDK 3610static libs as well as individual SPDK shared libs corresponding to 3611each of the SPDK static ones. Although the production of the shared 3612libs conforms with conventional version naming practices, such naming 3613does not at this time confer any SPDK ABI compatibility claims. 3614 3615### bdev 3616 3617spdk_bdev_alias_del_all() was added to delete all alias from block device. 3618 3619A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev 3620Framework. The module initially uses a software AESNI CBC cipher with experimental support for the 3621Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work 3622may include additional ciphers as well as consideration for authentication. 3623 3624The RAID virtual bdev module is now always enabled by default. The configure --with-raid and 3625--without-raid options are now ignored and deprecated and will be removed in the next release. 3626 3627Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer. 3628See the new [set_bdev_qos_limit](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit) 3629documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at 363018.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both 3631bandwidth and IOPS limits. 3632 3633spdk_bdev_config_json() and corresponding `get_bdevs_config` RPC was removed. 3634 3635### Environment Abstraction Layer and Event Framework 3636 3637The size parameter of spdk_mem_map_translate is now a pointer. This allows the 3638function to report back the actual size of the translation relative to the original 3639request made by the user. 3640 3641A new structure spdk_mem_map_ops has been introduced to hold memory map related 3642callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc 3643in lieu of the notify callback. 3644 3645### DPDK 18.08 3646 3647The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically 3648utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1. 3649 3650Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup, 3651but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try 3652to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using 3653only as many hugepages as it is really needed. 3654 3655Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed 3656to be physically contiguous. 3657 3658### I/OAT 3659 3660I/OAT driver can now reinitialize I/OAT channels after encountering DMA errors. 3661 3662### iscsi target 3663 3664Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC 3665method for CHAP authentication in discovery sessions have been changed to 3666align with `construct_target_node` RPC method. Old names are still usable 3667but will be removed in future release. 3668 3669`set_iscsi_discovery_auth` and `set_iscsi_target_node_auth` RPC methods have 3670been added to set CHAP authentication for discovery sessions and existing 3671target nodes, respectively. 3672 3673The SPDK iSCSI target supports an AuthFile which can be used to load CHAP 3674shared secrets when the iSCSI target starts. SPDK previously provided a 3675default location for this file (`/usr/local/etc/spdk/auth.conf`) if none was 3676specified. This default has been removed. Users must now explicitly specify 3677the location of this file to load CHAP shared secrets from a file, or use 3678the related iSCSI RPC methods to add them at runtime. 3679 3680### iscsi initiator 3681 3682The SPDK iSCSI initiator is no longer considered experimental and becomes 3683a first-class citizen among bdev modules. The basic usage has been briefly 3684described in the bdev user guide: [iSCSI bdev](https://spdk.io/doc/bdev.html#bdev_config_iscsi) 3685 3686### Miscellaneous 3687 3688The SPDK application framework can now parse long name command line parameters. 3689Most single-character parameters have a long name equivalent now. See the 3690[Command Line Parameters](https://spdk.io/doc/app_overview.html) documentation 3691for details or use the `--help` command line parameter to list all available 3692params. 3693 3694bdevperf `-s` param (io size) was renamed to `-o` as `-s` had been already 3695used by existing apps for memory size. 3696 3697bdevio can now accept all SPDK command line parameters. The config now has to 3698be provided with `-c` or `--config` param. 3699 3700The following ioat/perf and nvme/perf parameters were renamed as well: 3701 `-s` (io size) to `-o` 3702 `-d` (mem size) to `-s` 3703 3704The ReactorMask config file parameter has been deprecated. Users should 3705use the -m or --cpumask command line option to specify the CPU core mask 3706for the application. 3707 3708Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt 3709and vhost. Config file pathnames may now only be specified using the 3710-c command line option. 3711 3712Users may no longer set DPDK_DIR in their environment to specify the 3713location of the DPDK installation used to build SPDK. Using DPDK_DIR 3714has not been the documented nor recommended way to specify the DPDK 3715location for several releases, but removing it ensures no unexpected 3716surprises for users who may have DPDK_DIR defined for other reasons. 3717Users should just use the "configure" script to specify the DPDK 3718location before building SPDK. 3719 3720Although we know that many developers still use Python 2 we are officially 3721switching to Python3 with requirement that all new code must be valid also 3722for Python 2 up to the EOL which is year 2020. 3723 3724Invoking interpreter explicitly is forbidden for executable scripts. There 3725is no need to use syntax like "python ./scripts/rpc.py". All executable 3726scripts must contain proper shebang pointing to the right interpreter. 3727Scripts without shebang mustn't be executable. 3728 3729A Python script has been added to enable conversion of old INI config file 3730to new JSON-RPC config file format. This script can be found at 3731scripts/config_converter.py. Example how this script can be used: 3732~~~{.sh} 3733cat old_format.ini | scripts/config_converter.py > new_json_format.json 3734~~~ 3735 3736### Sock 3737 3738Two additional parameters were added to spdk_sock_get_addr() for the server 3739port and client port. These parameters are named "sport" and "cport" 3740respectively. 3741 3742### Virtio 3743 3744The following RPC commands have been deprecated: 3745 3746- construct_virtio_user_scsi_bdev 3747- construct_virtio_pci_scsi_bdev 3748- construct_virtio_user_blk_bdev 3749- construct_virtio_pci_blk_bdev 3750- remove_virtio_scsi_bdev 3751 3752The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev` 3753command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev` 3754was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI 3755devices. 3756 3757### Blobfs 3758 3759spdk_file_get_id() returning unique ID for the file was added. 3760 3761### JSON 3762 3763Added jsonrpc-client C library intended for issuing RPC commands from applications. 3764 3765Added API enabling iteration over JSON object: 3766 3767- spdk_json_find() 3768- spdk_json_find_string() 3769- spdk_json_find_array() 3770- spdk_json_object_first() 3771- spdk_json_array_first() 3772- spdk_json_next() 3773 3774### Blobstore 3775 3776Blobstore I/O operations are now based on io_units, instead of blobstore page size. 3777The io_unit size is now the same as the underlying block device's block size. 3778Logical volumes built on a block device with 512B block size can now be used as boot devices 3779in QEMU. 3780 3781### SPDKCLI 3782 3783The SPDKCLI interactive command tool for managing SPDK is no longer considered experimental. 3784Support for the iSCSI and NVMe-oF targets has been added. 3785 3786## v18.07: Raid, Infrastructure Improvements, Bug Fixes 3787 3788### bdev 3789 3790A new public header file bdev_module.h has been introduced to facilitate the 3791development of new bdev modules. This header includes an interface for the 3792spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of 3793multiple virtual bdevs on top of a single base bdev and should act as the 3794primary API for module authors. 3795 3796spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide 3797options. 3798 3799A mechanism for handling out of memory condition errors (ENOMEM) returned from 3800I/O submission requests at the bdev layer has been added. See 3801spdk_bdev_queue_io_wait(). 3802 3803The spdk_bdev_get_io_stat() function now returns cumulative totals instead of 3804resetting on each call. This allows multiple callers to query I/O statistics 3805without conflicting with each other. Existing users will need to adjust their 3806code to record the previous I/O statistics to calculate the delta between calls. 3807 3808I/O queue depth tracking and samples options have been added. See 3809spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and 3810spdk_bdev_set_qd_sampling_period(). 3811 3812### RAID module 3813 3814A new bdev module called "raid" has been added as experimental module which 3815aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note 3816that vhost will not work with this module because it does not yet have support 3817for multi-element io vectors. 3818 3819### Log 3820 3821The debug log component flag available on several SPDK applications has been 3822renamed from `-t` to `-L` to prevent confusion with tracepoints and to allow the 3823option to be added to tools that already use `-t` to mean something else. 3824 3825### Blobstore 3826 3827A new function, spdk_bs_dump(), has been added that dumps all of the contents of 3828a blobstore to a file pointer. This includes the metadata and is very useful for 3829debugging. 3830 3831Two new operations have been added for thin-provisioned blobs. 3832spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions 3833of the blob and populate them with the correct data by reading from the backing 3834blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate 3835clusters that correspond to data in the blob's immediate parent. Clusters 3836allocated to grandparents or that aren't allocated at all will remain 3837thin-provisioned. 3838 3839### BlobFS 3840 3841Changed the return type of spdk_file_truncate() from void to int to allow the 3842propagation of `ENOMEM` errors. 3843 3844### NVMe Driver 3845 3846The new API functions spdk_nvme_qpair_add_cmd_error_injection() and 3847spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error 3848emulation. Users can set a specified command to fail with a particular error 3849status. 3850 3851Changed the name `timeout_sec` parameter to `timeout_us` in 3852spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from 3853uint32_t to uint64_t. This will give users more fine-grained control over the 3854timeout period. 3855 3856Basic support for Open Channel SSDs was added. See nvme_ocssd.h 3857 3858### NVMe Over Fabrics 3859 3860The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback 3861as a parameter. 3862 3863spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs. 3864 3865spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed 3866number of namespaces for a given subsystem. 3867 3868### Build System 3869 3870The build system now generates a combined shared library (libspdk.so) that may 3871be used in place of the individual static libraries (libspdk_*.a). The combined 3872library includes all components of SPDK and is intended to make linking against 3873SPDK easier. The static libraries are also still provided for users that prefer 3874to link only the minimal set of components required. 3875 3876### git pre-commit and pre-push hooks 3877 3878The pre-commit hook will run `scripts/check_format.sh` and verify there are no 3879formatting errors before allowing `git commit` to run. The pre-push hook runs 3880`make CONFIG_WERROR=y` with and without `CONFIG_DEBUG=y` using both the gcc and 3881clang compiler before allowing `git push` to run. Following each DEBUG build 3882`test/unit/unittest.sh` is run and verified. Results are recorded in the 3883`make.log` file. 3884 3885To enable type: 'git config core.hooksPath .githooks'. To override after 3886configuration use the `git --no-verify` flag. 3887 3888### RPC 3889 3890The `start_nbd_disk` RPC method now returns the path to the kernel NBD device node 3891rather than always returning `true`. 3892 3893### DPDK 18.05 3894 3895The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports 3896dynamic memory allocation, but due to some issues found after the DPDK 18.05 release, 3897that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use 3898the legacy memory allocation model. The plan is to enable dynamic memory allocation 3899after the DPDK 18.08 release which should fix these issues. 3900 3901### Environment Abstraction Layer and Event Framework 3902 3903The spdk_mem_map_translate() function now takes a size parameter to indicate the size of 3904the memory region. This can be used by environment implementations to validate the 3905requested translation. 3906 3907The I/O Channel implementation has been moved to its own library - lib/thread. The 3908public API that was previously in spdk/io_channel.h is now in spdk/thread.h The 3909file spdk/io_channel.h remains and includes spdk/thread.h. 3910 3911spdk_reactor_get_tsc_stats was added to return interesting statistics for each 3912reactor. 3913 3914### IOAT 3915 3916IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable 3917option with "Yes" in `[Ioat]` section of the configuration file. The Disable option is 3918now deprecated and will be removed in a future release. 3919 3920## v18.04: Logical Volume Snapshot/Clone, iSCSI Initiator, Bdev QoS, VPP Userspace TCP/IP 3921 3922### vhost 3923 3924The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the 3925DPDK rte_vhost vulnerability [CVE-2018-1059](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1059). 3926Please see this [security advisory](https://access.redhat.com/security/cve/cve-2018-1059) 3927for additional information on the DPDK vulnerability. 3928 3929Workarounds have been added to ensure vhost compatibility with QEMU 2.12. 3930 3931EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the 3932[vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3933 3934### Unified Target Application 3935 3936A new unified SPDK target application, `spdk_tgt`, has been added. This application combines the 3937functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target, 3938and vhost target. The new application can be managed through the existing configuration file and 3939[JSON-RPC](http://www.spdk.io/doc/jsonrpc.html) methods. 3940 3941### Env 3942 3943spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk(). 3944 3945New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added. 3946These new functions have a `flags` parameter that allows the user to specify whether the allocated 3947memory needs to be suitable for DMA and whether it should be shared across processes with the same 3948shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will 3949eventually be deprecated and removed. 3950 3951### Bdev 3952 3953A new optional bdev module interface function, `init_complete`, has been added to notify bdev modules 3954when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require 3955notification that the set of initialization examine() calls is complete. 3956 3957The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with 3958the spdk_bdev_get_uuid() function. 3959 3960Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the 3961[set_bdev_qos_limit_iops](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit_iops) documentation 3962for more details. 3963 3964### RPC 3965 3966The `[Rpc]` configuration file section, which was deprecated in v18.01, has been removed. 3967Users should switch to the `-r` command-line parameter instead. 3968 3969The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as 3970needed; previously, the response was limited to 32 kilobytes. 3971 3972### SPDKCLI 3973 3974EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available. 3975See the [SPDKCLI](http://www.spdk.io/doc/spdkcli.html) documentation for more details. 3976 3977### NVMe Driver 3978 3979EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is 3980experimental pending a functional allocator to free and reallocate CMB buffers. 3981 3982spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available. 3983 3984New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns() 3985have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if 3986a namespace ID is active. 3987 3988### NVMe-oF Target 3989 3990Namespaces may now be assigned unique identifiers via new optional `eui64` and `nguid` parameters 3991to the `nvmf_subsystem_add_ns` RPC method. Additionally, the NVMe-oF target automatically exposes 3992the backing bdev's UUID as the namespace UUID when available. 3993 3994spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion. 3995 3996### Blobstore 3997 3998A number of functions have been renamed: 3999 4000- spdk_bs_io_write_blob() => spdk_blob_io_write() 4001- spdk_bs_io_read_blob() => spdk_blob_io_read() 4002- spdk_bs_io_writev_blob() => spdk_blob_io_writev() 4003- spdk_bs_io_readv_blob() => spdk_blob_io_readv() 4004- spdk_bs_io_unmap_blob() => spdk_blob_io_unmap() 4005- spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes() 4006 4007The old names still exist but are deprecated. They will be removed in the v18.07 release. 4008 4009spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O 4010are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still 4011required to sync the updated metadata to disk. 4012 4013### Logical Volumes 4014 4015A new `destroy_lvol_bdev` RPC method to delete logical volumes has been added. 4016 4017Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination. 4018 4019New Snapshot and Clone functionalities have been added. User may create Snapshots of existing Lvols 4020and Clones of existing Snapshots. 4021See the [lvol snapshots](http://www.spdk.io/doc/logical_volumes.html#lvol_snapshots) documentation 4022for more details. 4023 4024Resizing logical volumes is now supported via the `resize_lvol_bdev` RPC method. 4025 4026### Lib 4027 4028A set of changes were made in the SPDK's lib code altering 4029instances of calls to `exit()` and `abort()` to return a failure instead 4030wherever reasonably possible. 4031 4032spdk_app_start() no longer exit()'s on an internal failure, but 4033instead returns a non-zero error status. 4034 4035spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid 4036option, but instead returns SPDK_APP_PARSE_ARGS_HELP and 4037SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS 4038on success. 4039 4040spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07. 4041 4042### I/O Channels 4043 4044The prototype for spdk_poller_fn() has been modified; it now returns a value indicating 4045whether or not the poller did any work. Existing pollers will need to be updated to 4046return a value. 4047 4048### iSCSI Target 4049 4050The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace 4051TCP/IP stack. See the [iSCSI VPP documentation](http://www.spdk.io/doc/iscsi.html#vpp) for more 4052details. 4053 4054### iSCSI initiator 4055 4056An iSCSI initiator bdev module has been added to SPDK. This module should be considered 4057experimental pending additional features and tests. More details can be found in 4058lib/bdev/iscsi/README. 4059 4060### PMDK 4061 4062The persistent memory (PMDK) bdev module is now enabled using `--with-pmdk` instead of 4063`--with-nvml`. This reflects the renaming of the persistent memory library from NVML to 4064PMDK. 4065 4066### Virtio Block driver 4067 4068A userspace driver for Virtio Block devices has been added. It was built on top of the 4069[Virtio](http://www.spdk.io/doc/virtio.html) library and can be managed similarly to 4070the Virtio SCSI driver. See the 4071[Virtio Block](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_blk) reference for 4072more information. 4073 4074### Virtio with 2MB hugepages 4075 4076The previous 1GB hugepage limitation has now been lifted. A new `-g` command-line option 4077enables SPDK Virtio to work with 2MB hugepages. 4078See [2MB hugepages](http://www.spdk.io/doc/virtio.html#virtio_2mb) for details. 4079 4080## v18.01: Blobstore Thin Provisioning 4081 4082### Build System 4083 4084The build system now includes a `make install` rule, including support for the common 4085`DESTDIR` and `prefix` variables as used in other build systems. Additionally, the prefix 4086may be set via the configure `--prefix` option. Example: `make install prefix=/usr`. 4087 4088### RPC 4089 4090A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/tmp/spdk.sock. 4091A -r option command line option has been added to enable an alternative UNIX domain socket location, 4092or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file 4093section is now deprecated and will be removed in the v18.04 release. 4094 4095### I/O Channels 4096 4097spdk_poller_register() and spdk_poller_unregister() were moved from the event 4098framework (include/spdk/event.h) to the I/O channel library 4099(include/spdk/io_channel.h). This allows code that doesn't depend on the event 4100framework to request registration and unregistration of pollers. 4101 4102spdk_for_each_channel() now allows asynchronous operations during iteration. 4103Instead of immediately continuing the interaction upon returning from the iteration 4104callback, the user must call spdk_for_each_channel_continue() to resume iteration. 4105 4106### Block Device Abstraction Layer (bdev) 4107 4108The poller abstraction was removed from the bdev layer. There is now a general purpose 4109abstraction for pollers available in include/spdk/io_channel.h 4110 4111### Lib 4112 4113A set of changes were made in the SPDK's lib code altering, 4114instances of calls to `exit()` and `abort()` to return a failure instead 4115wherever reasonably possible. This has resulted in return type changes of 4116the API for: 4117 4118- spdk_env_init() from type `void` to `int`. 4119- spdk_mem_map_init() from type `void` to `int`. 4120 4121Applications making use of these APIs should be modified to check for 4122a non-zero return value instead of relying on them to fail without return. 4123 4124### NVMe Driver 4125 4126SPDK now supports hotplug for vfio-attached devices. But there is one thing keep in mind: 4127Only physical removal events are supported; removing devices via the sysfs `remove` file will not work. 4128 4129### NVMe-oF Target 4130 4131Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available 4132cores round-robin. The "Core" option in the configuration file has been removed. 4133 4134### Blobstore 4135 4136A number of functions have been renamed: 4137 4138- spdk_bs_md_resize_blob() => spdk_blob_resize() 4139- spdk_bs_md_sync_blob() => spdk_blob_sync_md() 4140- spdk_bs_md_close_blob() => spdk_blob_close() 4141- spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names() 4142- spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value() 4143- spdk_blob_md_set_xattr() => spdk_blob_set_xattr() 4144- spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr() 4145- spdk_bs_md_create_blob() => spdk_bs_create_blob() 4146- spdk_bs_md_open_blob() => spdk_bs_open_blob() 4147- spdk_bs_md_delete_blob() => spdk_bs_delete_blob() 4148- spdk_bs_md_iter_first() => spdk_bs_iter_first() 4149- spdk_bs_md_iter_next() => spdk_bs_iter_next() 4150 4151The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument 4152rather than struct spdk_blob **. 4153 4154The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument 4155rather than struct spdk_blob **. 4156 4157Thin provisioning support has been added to the blobstore. It can be enabled by setting the 4158`thin_provision` flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext(). 4159 4160### NBD device 4161 4162The NBD application (test/lib/bdev/nbd) has been removed; Same functionality can now be 4163achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method. 4164See the [GPT](http://www.spdk.io/doc/bdev.html#bdev_config_gpt) documentation for more details. 4165 4166### FIO plugin 4167 4168SPDK `fio_plugin` now supports FIO 3.3. The support for previous FIO 2.21 has been dropped, 4169although it still remains to work for now. The new FIO contains huge amount of bugfixes and 4170it's recommended to do an update. 4171 4172### Virtio library 4173 4174Previously a part of the bdev_virtio module, now a separate library. Virtio is now available 4175via `spdk_internal/virtio.h` file. This is an internal interface to be used when implementing 4176new Virtio backends, namely Virtio-BLK. 4177 4178### iSCSI 4179 4180The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated 4181to an epoll/kqueue descriptor on the master core when idle. 4182 4183## v17.10: Logical Volumes 4184 4185### New dependencies 4186 4187libuuid was added as new dependency for logical volumes. 4188 4189libnuma is now required unconditionally now that the DPDK submodule has been updated to DPDK 17.08. 4190 4191### Block Device Abstraction Layer (bdev) 4192 4193An [fio](http://github.com/axboe/fio) plugin was added that can route 4194I/O to the bdev layer. See the [plugin documentation](https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin/) 4195for more information. 4196 4197spdk_bdev_unmap() was modified to take an offset and a length in bytes as 4198arguments instead of requiring the user to provide an array of SCSI 4199unmap descriptors. This limits unmaps to a single contiguous range. 4200 4201spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out. 4202If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using 4203write commands. 4204 4205New API functions that accept I/O parameters in units of blocks instead of bytes 4206have been added: 4207 4208- spdk_bdev_read_blocks(), spdk_bdev_readv_blocks() 4209- spdk_bdev_write_blocks(), spdk_bdev_writev_blocks() 4210- spdk_bdev_write_zeroes_blocks() 4211- spdk_bdev_unmap_blocks() 4212 4213The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be 4214retried later. 4215 4216### Linux AIO bdev 4217 4218The AIO bdev now allows the user to override the auto-detected block size. 4219 4220### NVMe driver 4221 4222The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field. 4223NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take 4224into account when splitting I/O requests. 4225 4226The HotplugEnable option in `[Nvme]` sections of the configuration file is now 4227"No" by default. It was previously "Yes". 4228 4229The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the 4230NVMe Controller associated with a given namespace. 4231 4232The NVMe library now allows the user to specify a host identifier when attaching 4233to a controller. The host identifier is used as part of the Reservations feature, 4234as well as in the NVMe-oF Connect command. The default host ID is also now a 4235randomly-generated UUID, and the default host NQN uses the host ID to generate 4236a UUID-based NQN. 4237 4238spdk_nvme_connect() was added to allow the user to connect directly to a single 4239NVMe or NVMe-oF controller. 4240 4241### NVMe-oF Target (nvmf_tgt) 4242 4243The NVMe-oF target no longer requires any in-capsule data buffers to run, and 4244the feature is now entirely optional. Previously, at least 4 KiB in-capsule 4245data buffers were required. 4246 4247NVMe-oF subsystems have a new configuration option, AllowAnyHost, to control 4248whether the host NQN whitelist is enforced when accepting new connections. 4249If no Host options have been specified and AllowAnyHost is disabled, the 4250connection will be denied; this is a behavior change from previous releases, 4251which allowed any host NQN to connect if the Host list was empty. 4252AllowAnyHost is disabled by default. 4253 4254NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number 4255of namespaces per subsystem is no longer limited. 4256 4257The NVMe-oF target now supports the Write Zeroes command. 4258 4259### Environment Abstraction Layer 4260 4261A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide 4262additional clarity when constructing spdk_mempools. Previously, -1 could be 4263passed and the library would choose a reasonable default, but this new value 4264makes it explicit that the default is being used. 4265 4266### Blobstore 4267 4268The blobstore super block now contains a bstype field to identify the type of the blobstore. 4269Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load(). 4270 4271spdk_bs_destroy() was added to allow destroying blobstore on device 4272with an initialized blobstore. 4273 4274spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable 4275scattered payloads. 4276 4277A CLI tool for blobstore has been added, allowing basic operations through either command 4278line or shell interface. See the [blobcli](https://github.com/spdk/spdk/tree/master/examples/blob/cli) 4279documentation for more details. 4280 4281### Event Framework 4282 4283The ability to set a thread name, previously only used by the reactor code, is 4284now part of the spdk_thread_allocate() API. Users may specify a thread name 4285which will show up in tools like `gdb`. 4286 4287### Log 4288 4289The spdk_trace_dump() function now takes a new parameter to allow the caller to 4290specify an output file handle (stdout or stderr, for example). 4291 4292### Logical Volumes 4293 4294Logical volumes library built on top of SPDK blobstore has been added. 4295It is possible to create logical volumes on top of other devices using RPC. 4296 4297See the [logical volumes](http://www.spdk.io/doc/logical_volumes.html) documentation for more information. 4298 4299### Persistent Memory 4300 4301A new persistent memory bdev type has been added. 4302The persistent memory block device is built on top of [libpmemblk](http://pmem.io/nvml/libpmemblk/). 4303It is possible to create pmem devices on top of pmem pool files using RPC. 4304 4305See the [Pmem Block Device](http://www.spdk.io/doc/bdev.html#bdev_config_pmem) documentation for more information. 4306 4307### Virtio SCSI driver 4308 4309A userspace driver for Virtio SCSI devices has been added. 4310The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application. 4311 4312See the [Virtio SCSI](http://www.spdk.io/doc/virtio.html) documentation and 4313[Getting Started](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_scsi) guide for more information. 4314 4315### Vhost target 4316 4317The vhost target application now supports live migration between QEMU instances. 4318 4319## v17.07: Build system improvements, userspace vhost-blk target, and GPT bdev 4320 4321### Build System 4322 4323A `configure` script has been added to simplify the build configuration process. 4324The existing CONFIG file and `make CONFIG_...` options are also still supported. 4325Run `./configure --help` for information about available configuration options. 4326 4327A DPDK submodule has been added to make building SPDK easier. If no `--with-dpdk` 4328option is specified to configure, the SPDK build system will automatically build a 4329known-good configuration of DPDK with the minimal options enabled. See the Building 4330section of README.md for more information. 4331 4332A [Vagrant](https://www.vagrantup.com/) setup has been added to make it easier to 4333develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant 4334section of README.md for more information. 4335 4336### Userspace vhost-blk target 4337 4338The vhost library and example app have been updated to support the vhost-blk 4339protocol in addition to the existing vhost-scsi protocol. 4340See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 4341 4342### Block device abstraction layer (bdev) 4343 4344A GPT virtual block device has been added, which automatically exposes GPT partitions 4345with a special SPDK-specific partition type as bdevs. 4346See the [GPT bdev documentation](http://www.spdk.io/doc/bdev.md#bdev_config_gpt) for 4347more information. 4348 4349### NVMe driver 4350 4351The NVMe driver has been updated to support recent Intel SSDs, including the Intel® 4352Optane™ SSD DC P4800X series. 4353 4354A workaround has been added for devices that failed to recognize register writes 4355during controller reset. 4356 4357The NVMe driver now allocates request tracking objects on a per-queue basis. The 4358number of requests allowed on an I/O queue may be set during `spdk_nvme_probe()` by 4359modifying `io_queue_requests` in the opts structure. 4360 4361The SPDK NVMe `fio_plugin` has been updated to support multiple threads (`numjobs`). 4362 4363spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override 4364controller-level options for each individual I/O queue pair. 4365Existing callers with qprio == 0 can be updated to: 4366~~~ 4367... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0); 4368~~~ 4369Callers that need to specify a non-default qprio should be updated to: 4370~~~ 4371struct spdk_nvme_io_qpair_opts opts; 4372spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts)); 4373opts.qprio = SPDK_NVME_QPRIO_...; 4374... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts)); 4375~~~ 4376 4377### Environment Abstraction Layer 4378 4379The environment abstraction layer has been updated to include several new functions 4380in order to wrap additional DPDK functionality. See `include/spdk/env.h` for the 4381current set of functions. 4382 4383### SPDK Performance Analysis with Intel® VTune™ Amplifier 4384 4385Support for SPDK performance analysis has been added to Intel® VTune™ Amplifier 2018. 4386 4387This analysis provides: 4388 4389- I/O performance monitoring (calculating standard I/O metrics like IOPS, throughput, etc.) 4390- Tuning insights on the interplay of I/O and compute devices by estimating how many cores 4391 would be reasonable to provide for SPDK to keep up with a current storage workload. 4392 4393See the VTune Amplifier documentation for more information. 4394 4395## v17.03: Blobstore and userspace vhost-scsi target 4396 4397### Blobstore and BlobFS 4398 4399The blobstore is a persistent, power-fail safe block allocator designed to be 4400used as the local storage system backing a higher-level storage service. 4401See the [blobstore documentation](http://www.spdk.io/doc/blob.html) for more details. 4402 4403BlobFS adds basic filesystem functionality like filenames on top of the blobstore. 4404This release also includes a RocksDB Env implementation using BlobFS in place of the 4405kernel filesystem. 4406See the [BlobFS documentation](http://www.spdk.io/doc/blobfs.html) for more details. 4407 4408### Userspace vhost-scsi target 4409 4410A userspace implementation of the QEMU vhost-scsi protocol has been added. 4411The vhost target is capable of exporting SPDK bdevs to QEMU-based VMs as virtio devices. 4412See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 4413 4414### Event framework 4415 4416The overhead of the main reactor event loop was reduced by optimizing the number of 4417calls to spdk_get_ticks() per iteration. 4418 4419### NVMe library 4420 4421The NVMe library will now automatically split readv/writev requests with scatter-gather 4422lists that do not map to valid PRP lists when the NVMe controller does not natively 4423support SGLs. 4424 4425The `identify` and `perf` NVMe examples were modified to add a consistent format for 4426specifying remote NVMe over Fabrics devices via the `-r` option. 4427This is implemented using the new `spdk_nvme_transport_id_parse()` function. 4428 4429### iSCSI Target 4430 4431The [Nvme] section of the configuration file was modified to remove the `BDF` directive 4432and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 4433devices can now be specified as the backing block device. A script to generate an 4434entire [Nvme] section based on the local NVMe devices attached was added at 4435`scripts/gen_nvme.sh`. 4436 4437### NVMe-oF Target 4438 4439The [Nvme] section of the configuration file was modified to remove the `BDF` directive 4440and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 4441devices can now be specified as the backing block device. A script to generate an 4442entire [Nvme] section based on the local NVMe devices attached was added at 4443`scripts/gen_nvme.sh`. 4444 4445## v16.12: NVMe over Fabrics host, hotplug, and multi-process 4446 4447### NVMe library 4448 4449The NVMe library has been changed to create its own request memory pool rather than 4450requiring the user to initialize the global `request_mempool` variable. Apps can be 4451updated by simply removing the initialization of `request_mempool`. Since the NVMe 4452library user no longer needs to know the size of the internal NVMe request 4453structure to create the pool, the `spdk_nvme_request_size()` function was also removed. 4454 4455The `spdk_nvme_ns_cmd_deallocate()` function was renamed and extended to become 4456`spdk_nvme_ns_cmd_dataset_management()`, which allows access to all of the NVMe 4457Dataset Management command's parameters. Existing callers can be updated to use 4458`spdk_nvme_ns_cmd_dataset_management()` with `SPDK_NVME_DSM_ATTR_DEALLOCATE` as the 4459`type` parameter. 4460 4461The NVMe library SGL callback prototype has been changed to return virtual addresses 4462rather than physical addresses. Callers of `spdk_nvme_ns_cmd_readv()` and 4463`spdk_nvme_ns_cmd_writev()` must update their `next_sge_fn` callbacks to match. 4464 4465The NVMe library now supports NVMe over Fabrics devices in addition to the existing 4466support for local PCIe-attached NVMe devices. For an example of how to enable 4467NVMe over Fabrics support in an application, see `examples/nvme/identify` and 4468`examples/nvme/perf`. 4469 4470Hot insert/remove support for NVMe devices has been added. To enable NVMe hotplug 4471support, an application should call the `spdk_nvme_probe()` function on a regular 4472basis to probe for new devices (reported via the existing `probe_cb` callback) and 4473removed devices (reported via a new `remove_cb` callback). Hotplug is currently 4474only supported on Linux with the `uio_pci_generic` driver, and newly-added NVMe 4475devices must be bound to `uio_pci_generic` by an external script or tool. 4476 4477Multiple processes may now coordinate and use a single NVMe device simultaneously 4478using [DPDK Multi-process Support](http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html). 4479 4480### NVMe over Fabrics target (`nvmf_tgt`) 4481 4482The `nvmf_tgt` configuration file format has been updated significantly to enable 4483new features. See the example configuration file `etc/spdk/nvmf.conf.in` for 4484more details on the new and changed options. 4485 4486The NVMe over Fabrics target now supports virtual mode subsystems, which allow the 4487user to export devices from the SPDK block device abstraction layer as NVMe over 4488Fabrics subsystems. Direct mode (raw NVMe device access) is also still supported, 4489and a single `nvmf_tgt` may export both types of subsystems simultaneously. 4490 4491### Block device abstraction layer (bdev) 4492 4493The bdev layer now supports scatter/gather read and write I/O APIs, and the NVMe 4494blockdev driver has been updated to support scatter/gather. Apps can use the 4495new scatter/gather support via the `spdk_bdev_readv()` and `spdk_bdev_writev()` 4496functions. 4497 4498The bdev status returned from each I/O has been extended to pass through NVMe 4499or SCSI status codes directly in cases where the underlying device can provide 4500a more specific status code. 4501 4502A Ceph RBD (RADOS Block Device) blockdev driver has been added. This allows the 4503`iscsi_tgt` and `nvmf_tgt` apps to export Ceph RBD volumes as iSCSI LUNs or 4504NVMe namespaces. 4505 4506### General changes 4507 4508`libpciaccess` has been removed as a dependency and DPDK PCI enumeration is 4509used instead. Prior to DPDK 16.07 enumeration by class code was not supported, 4510so for earlier DPDK versions, only Intel SSD DC P3x00 devices will be discovered 4511by the NVMe library. 4512 4513The `env` environment abstraction library has been introduced, and a default 4514DPDK-based implementation is provided as part of SPDK. The goal of the `env` 4515layer is to enable use of alternate user-mode memory allocation and PCI access 4516libraries. See `doc/porting.md` for more details. 4517 4518The build process has been modified to produce all of the library files in the 4519`build/lib` directory. This is intended to simplify the use of SPDK from external 4520projects, which can now link to SPDK libraries by adding the `build/lib` directory 4521to the library path via `-L` and linking the SPDK libraries by name (for example, 4522`-lspdk_nvme -lspdk_log -lspdk_util`). 4523 4524`nvmf_tgt` and `iscsi_tgt` now have a JSON-RPC interface, which allows the user 4525to query and modify the configuration at runtime. The RPC service is disabled by 4526default, since it currently does not provide any authentication or security 4527mechanisms; it should only be enabled on systems with controlled user access 4528behind a firewall. An example RPC client implemented in Python is provided in 4529`scripts/rpc.py`. 4530 4531## v16.08: iSCSI target, NVMe over Fabrics maturity 4532 4533This release adds a userspace iSCSI target. The iSCSI target is capable of exporting 4534NVMe devices over a network using the iSCSI protocol. The application is located 4535in app/iscsi_tgt and a documented configuration file can be found at etc/spdk/spdk.conf.in. 4536 4537This release also significantly improves the existing NVMe over Fabrics target. 4538 4539- The configuration file format was changed, which will require updates to 4540 any existing nvmf.conf files (see `etc/spdk/nvmf.conf.in`): 4541 - `SubsystemGroup` was renamed to `Subsystem`. 4542 - `AuthFile` was removed (it was unimplemented). 4543 - `nvmf_tgt` was updated to correctly recognize NQN (NVMe Qualified Names) 4544 when naming subsystems. The default node name was changed to reflect this; 4545 it is now "nqn.2016-06.io.spdk". 4546 - `Port` and `Host` sections were merged into the `Subsystem` section 4547 - Global options to control max queue depth, number of queues, max I/O 4548 size, and max in-capsule data size were added. 4549 - The Nvme section was removed. Now a list of devices is specified by 4550 bus/device/function directly in the Subsystem section. 4551 - Subsystems now have a Mode, which can be Direct or Virtual. This is an attempt 4552 to future-proof the interface, so the only mode supported by this release 4553 is "Direct". 4554- Many bug fixes and cleanups were applied to the `nvmf_tgt` app and library. 4555- The target now supports discovery. 4556 4557This release also adds one new feature and provides some better examples and tools 4558for the NVMe driver. 4559 4560- The Weighted Round Robin arbitration method is now supported. This allows 4561 the user to specify different priorities on a per-I/O-queue basis. To 4562 enable WRR, set the `arb_mechanism` field during `spdk_nvme_probe()`. 4563- A simplified "Hello World" example was added to show the proper way to use 4564 the NVMe library API; see `examples/nvme/hello_world/hello_world.c`. 4565- A test for measuring software overhead was added. See `test/lib/nvme/overhead`. 4566 4567## v16.06: NVMf userspace target 4568 4569This release adds a userspace NVMf (NVMe over Fabrics) target, conforming to the 4570newly-released NVMf 1.0/NVMe 1.2.1 specification. The NVMf target exports NVMe 4571devices from a host machine over the network via RDMA. Currently, the target is 4572limited to directly exporting physical NVMe devices, and the discovery subsystem 4573is not supported. 4574 4575This release includes a general API cleanup, including renaming all declarations 4576in public headers to include a `spdk` prefix to prevent namespace clashes with 4577user code. 4578 4579- NVMe 4580 - The `nvme_attach()` API was reworked into a new probe/attach model, which 4581 moves device detection into the NVMe library. The new API also allows 4582 parallel initialization of NVMe controllers, providing a major reduction in 4583 startup time when using multiple controllers. 4584 - I/O queue allocation was changed to be explicit in the API. Each function 4585 that generates I/O requests now takes a queue pair (`spdk_nvme_qpair *`) 4586 argument, and I/O queues may be allocated using 4587 `spdk_nvme_ctrlr_alloc_io_qpair()`. This allows more flexible assignment of 4588 queue pairs than the previous model, which only allowed a single queue 4589 per thread and limited the total number of I/O queues to the lowest number 4590 supported on any attached controller. 4591 - Added support for the Write Zeroes command. 4592 - `examples/nvme/perf` can now report I/O command latency from the 4593 the controller's viewpoint using the Intel vendor-specific read/write latency 4594 log page. 4595 - Added namespace reservation command support, which can be used to coordinate 4596 sharing of a namespace between multiple hosts. 4597 - Added hardware SGL support, which enables use of scattered buffers that 4598 don't conform to the PRP list alignment and length requirements on supported 4599 NVMe controllers. 4600 - Added end-to-end data protection support, including the ability to write and 4601 read metadata in extended LBA (metadata appended to each block of data in the 4602 buffer) and separate metadata buffer modes. 4603 See `spdk_nvme_ns_cmd_write_with_md()` and `spdk_nvme_ns_cmd_read_with_md()` 4604 for details. 4605- IOAT 4606 - The DMA block fill feature is now exposed via the `ioat_submit_fill()` 4607 function. This is functionally similar to `memset()`, except the memory is 4608 filled with an 8-byte repeating pattern instead of a single byte like memset. 4609- PCI 4610 - Added support for using DPDK for PCI device mapping in addition to the 4611 existing libpciaccess option. Using the DPDK PCI support also allows use of 4612 the Linux VFIO driver model, which means that SPDK userspace drivers will work 4613 with the IOMMU enabled. Additionally, SPDK applications may be run as an 4614 unprivileged user with access restricted to a specific set of PCIe devices. 4615 - The PCI library API was made more generic to abstract away differences 4616 between the underlying PCI access implementations. 4617 4618## v1.2.0: IOAT user-space driver 4619 4620This release adds a user-space driver with support for the Intel I/O Acceleration Technology 4621(I/OAT, also known as "Crystal Beach") DMA offload engine. 4622 4623- IOAT 4624 - New user-space driver supporting DMA memory copy offload 4625 - Example programs `ioat/perf` and `ioat/verify` 4626 - Kernel-mode DMA engine test driver `kperf` for performance comparison 4627- NVMe 4628 - Per-I/O flags for Force Unit Access (FUA) and Limited Retry 4629 - Public API for retrieving log pages 4630 - Reservation register/acquire/release/report command support 4631 - Scattered payload support - an alternate API to provide I/O buffers via a sequence of callbacks 4632 - Declarations and `nvme/identify` support for Intel SSD DC P3700 series vendor-specific log pages and features 4633- Updated to support DPDK 2.2.0 4634 4635## v1.0.0: NVMe user-space driver 4636 4637This is the initial open source release of the Storage Performance Development Kit (SPDK). 4638 4639Features: 4640 4641- NVMe user-space driver 4642- NVMe example programs 4643 - `examples/nvme/perf` tests performance (IOPS) using the NVMe user-space driver 4644 - `examples/nvme/identify` displays NVMe controller information in a human-readable format 4645- Linux and FreeBSD support 4646