1# Changelog 2 3## v24.09: (Upcoming Release) 4 5### thread 6 7New function `spdk_interrupt_register_for_events()` build on top of `spdk_fd_group_add_for_events()`. 8See below for details. 9 10### util 11 12New function `spdk_fd_group_add_for_events()` was added alongside the existing `spdk_fd_group_add()`. 13Difference is that new API allows for specifying a set of events to be monitored instead of default 14SPDK_INTERRUPT_EVENT_IN. 15 16### nvmf 17 18Added public API 'spdk_nvmf_subsystem_set_cntlid_range' to set controller ID 19range for a subsystem. 20 21### event 22 23The `framework_get_reactors` RPC method supports getting pid and tid. 24 25### env_dpdk 26 27`spdk_get_tid` is added to get the tid of the current thread. 28 29## v24.05 30 31### accel 32 33`spdk_accel_get/set_opts()` has changed to act more like spdk_bdev's variant. 34 35### bdev 36 37Added `spdk_bdev_get_nvme_ctratt()` API to get controller attributes of bdev. 38 39### bdev_raid 40 41Added support for interleaved metadata. 42 43Added RAID1 I/O error handling. 44 45Examining a bdev with RAID superblock now searches for other base bdevs that may be present. 46 47Re-adding a removed base bdev with superblock will now begin rebuild on applicable RAID levels. 48 49Base bdevs can now be added back if were removed while in configuring state. 50 51RAID in configuring state will not be removed if the last base bdev is removed. 52 53### bdev_uring 54 55Added `bdev_uring_rescan` RPC to allow rescaning the size of uring bdev. 56 57### blobstore 58 59Added new API `spdk_bs_blob_shallow_copy()` to make a shallow copy from a blob to a blobstore device. Only clusters 60allocated to the blob will be written on the device. 61 62Added new APIs `spdk_bs_blob_set_parent()` and `spdk_bs_blob_set_external_parent()` to change the parent of an 63existing lvol. With these new API a thin provisioned blob can become a clone of an existing snapshot 64or of an existing external snapshot. 65 66### event 67 68SPDK applications can now start with `--wait-for-rpc` and JSON configuration provided at the same time. 69 70Struct `spdk_app_opts` was extended by `json_data` and `json_data_size` fields 71allowing user to provide a buffer with JSON config as alternative to providing JSON config file path 72(these fields are mutually exclusive). 73 74### init 75 76Function `spdk_subsystem_init_from_json_config()` is deprecated and will be removed in 24.09 release. 77Please use `spdk_subsystem_load_config()` instead. 78 79### iscsi 80 81Added `iscsi_get_stats` RPC method to get stat information about all active connections. 82 83Added `iscsi_enable_histogram` RPC method to enable or disable histogram for specified iSCSI target. 84 85Added `iscsi_get_histogram` RPC method to get histogram for specified iSCSI target. 86 87### keyring 88 89Added keyring library. Its purpose is to provide secure access to cryptographic keys from other 90libraries. Prior to being used, keys need to be attached to the keyring using one of the modules, 91which are responsible for retrieving keying material in a secure manner. Currently, there are two 92keyring modules implemented: `keyring_file` (keys are stored in files on a filesystem) and 93`keyring_linux` (based on Linux kernel keyring). The module interface is pluggable, so users can 94implement their own keyring modules. 95 96### lvol 97 98Added `bdev_lvol_start_shallow_copy` RPC to start a shallow copy of an lvol over a given bdev and 99`bdev_lvol_check_shallow_copy` RPC to get the status of the operation. 100These RPCs use the new API `spdk_lvol_shallow_copy()`. 101 102Added `bdev_lvol_set_parent` and `bdev_lvol_set_parent_bdev` RPC to change the parent of an existing lvol 103with the use of the new APIs `spdk_lvol_set_parent()` and `spdk_lvol_set_external_parent()`. 104 105### nvme 106 107Added `spdk_nvme_ctrlr_get_max_sges()` API to retrieve maximum number of SGEs per request 108for the given NVMe controller. 109 110The NVMe driver now supports in-band authentication using the DH-HMAC-CHAP protocol. To enable it, 111users need to specify keys in the `bdev_nvme_attach_controller` RPC. Additionally, it's possible to 112limit the allowed digests and Diffie-Hellman groups via `bdev_nvme_set_options`. 113 114### nvmf 115 116Added support for namespace masking using new C APIs `spdk_nvmf_ns_add_host()` and 117`spdk_nvmf_ns_remove_host()` and RPCs `nvmf_ns_add_host` and `nvmf_ns_remove_host`. 118Users must pass new `--no-auto-visible` parameter to `nvmf_subsystem_add_ns` RPC to allow 119namespace masking to be controlled by these new RPCs. 120 121Added `spdk_nvmf_subsystem_get_ana_state()` API to get current ANA state of 122a particular ANA group ID. 123 124Added support for enabling mDNS-based discovery of nvmf target for 125TCP transport with the addition of `nvmf_publish_mdns_prr` and 126`nvmf_stop_mdns_prr` RPCs. 127 128The NVMe-oF target now supports in-band authentication using the DH-HMAC-CHAP protocol. The target 129will request hosts to authenticate if they're configured to use DH-HMAC-CHAP keys. The keys can be 130set in the `nvmf_subsystem_add_host` RPC. Additionally, it's possible to limit the allowed digests 131and Diffie-Hellman groups via `nvmf_set_config`. 132 133### spdk_trace 134 135`spdk_trace` has learned how to use the most recent trace file in /dev/shm when 136the user has specified neither `-f` nor `-s` options. This is only available on 137Linux since FreeBSD does not mount have a /dev/shm mount. 138 139### thread 140 141Changed `spdk_iobuf_get/set_opts()` API to act more like spdk_bdev's variant. 142 143### trace 144 145Merged `struct spdk_trace_flags` and `struct spdk_trace_histories` into 146new `struct spdk_trace_file`. Also renamed `spdk_get_trace_histories_size()` 147to `spdk_get_trace_file_size()`. 148 149## v24.01 150 151### accel 152 153Added support for new operations for computing Data Integrity Field (DIF), 154`spdk_accel_submit_dif_verify()`, `spdk_accel_submit_dif_generate()` and 155`spdk_accel_submit_dif_generate_copy()`. 156 157Added `spdk_accel_get_opcode_name()` API that returns name of an operation. 158 159Added `spdk_accel_get_module()` API that returns pointer to a module. 160 161Added optional `init()` and `fini()` callbacks to `spdk_accel_driver` in order to allow 162drivers to initialize and clean up resources by the driver. 163 164### accel_error 165 166Added new `error` accel module that allows user to inject error to selected accel operations. 167 168### bdev 169 170Added `spdk_bdev_nvme_iov_passthru_md()` API to allow sending larger passthru commands. 171 172Added `max_rw_size` field to `spdk_bdev` structure allowing bdev modules to specify maximum 173size of an I/O. 174 175Added `iobuf_small_cache_size` and `iobuf_large_cache_size` parameters to 176`bdev_set_options` RPC making iobuf use in bdev configurable. 177 178### bdev_raid 179 180Added support for rebuild in raid1 and raid5f levels. 181 182Added `superblock` argument to `bdev_raid_create` RPC to allow writing out raid metadata to 183a superblock. It can be then re-created from examined base bdevs, without the need 184to issue `bdev_raid_create`. 185 186Added `bdev_raid_add_base_bdev` RPC to add a base bdevs to an existing raid bdev. 187 188Added `bdev_raid_set_options` RPC that changes options for raid bdev module, 189currently allows setting `process_window_size_kb` affecting background operations 190like rebuild. 191 192### blobstore 193 194Added `spdk_bs_grow_live()` and `spdk_bdev_update_bs_blockcnt()` API that can be used to 195increase size of blobstore filling the underlying device without first closing the blobstore. 196 197### env 198 199Added SPDK command line parameter `--no-huge`, which enables SPDK to run without hugepages. 200 201### event 202 203Added SPDK command line parameter `--interrupt-mode`, which enables to run SPDK in interrupt 204mode. Effective only if all components used in the application, support interrupt mode. 205 206### ftl 207 208Added `rpc_bdev_ftl_get_properties` and `bdev_ftl_set_property` RPC with matching API calls, 209to allow modification of FTL device properties. 210 211### idxd 212 213Added `spdk_idxd_submit_dif_check()` and `spdk_idxd_submit_dif_insert()` API to support 214new operations for computing Data Integrity Field (DIF). 215 216### json 217 218Added `spdk_json_write_uuid()`, `spdk_json_write_named_uuid()` and `spdk_json_decode_uuid()` 219to help with writing and decoding UUID. 220 221### nvme 222 223A new transport option `rdma_max_cq_size` was added to limit indefinite growth of CQ size. 224 225Added `spdk_nvme_ctrlr_cmd_iov_raw_with_md()` API to allow sending larger passthru commands. 226 227### nvmf 228 229Added `max_discard_size_kib` and `max_write_zeroes_size_kib` to `nvmf_create_subsystem` RPC to set the 230maximum discard size and maximum write zeroes size. 231 232Added new optional `--ana-state` (or shortly `-n`) parameter to `nvmf_subsystem_add_listener` RPC. 233 234Added public APIs `spdk_nvmf_subsystem_get_ana_reporting()` and `spdk_nvmf_subsystem_set_ana_state()`, 235replacing the internal functions `nvmf_subsystem_get_ana_reporting()` and `nvmf_subsystem_set_ana_state()` 236respectively. 237 238Added support for NVMe-oF referrals in NVMe-oF target with addition of 239`spdk_nvmf_tgt_add_referral()` and `spdk_nvmf_tgt_remove_referral()` API and `nvmf_discovery_add_referral`, 240`nvmf_discovery_remove_referral` and `nvmf_discovery_get_referrals` RPC. 241 242Added `spdk_nvmf_set_custom_ns_reservation_ops()` API allowing to register custom handlers for reservation 243operations. 244 245### scsi 246 247Added support for `SPDK_SBC_WRITE_SAME_10` and `SPDK_SBC_WRITE_SAME_16`. 248 249### thread 250 251Added `iobuf_get_stats` RPC and `spdk_iobuf_get_stats()` API to track iobuf use across components. 252 253### trace 254 255Added `spdk_trace_register_user_thread()` to initialize trace environment and 256`spdk_trace_unregister_user_thread()` to de-initialize trace environment 257for a user created thread. 258 259Modified `spdk_trace_init()` to take number of user created threads as a parameter. 260 261### ublk 262 263Added `ublk_recover_disk` RPC to support recovering after restarting ublk target. 264 265### vhost 266 267Added `caw_iov` field to struct `spdk_scsi_task` to support SBC-3 compare_and_write IO. 268 269Added parameter `delay` to `vhost_create_scsi_controller` RPC which allows user to defer starting 270the vhost-scsi controller until adding the scsi target is completed. 271 272Added `vhost_start_scsi_controller` RPC to start vhost-scsi controller, it could be used to support 273live recovery feature of vhost-scsi target. 274 275### rpc 276 277Functions `spdk_rpc_listen`, `spdk_rpc_accept` and `spdk_rpc_close` are deprecated and will be 278removed in 24.09 release. Please use `spdk_rpc_server_listen`, `spdk_rpc_server_accept` and 279`spdk_rpc_server_close` instead. 280 281## v23.09 282 283### accel 284 285Added `spdk_accel_get_buf_align()` API to get minimum buffer alignment to execute a given operation. 286 287Added `get_operation_info()` callback for accel_modules to return constraints for a given operation. 288 289### bdev 290 291Added `spdk_bdev_open_async()` API to allow asynchronous bdev open that waits for specified time 292until the block device appears. 293 294### bdev_nvme 295 296Added `bdev_nvme_enable_controller` and `bdev_nvme_disable_controller` RPC to enable and disable 297NVMe controller. For NVMe multipath `cntlid` is used to select only one NVMe-oF controller. 298 299Added `cntlid` field to `bdev_nvme_reset_controller` RPC to support resetting only one NVMe-oF 300controller in multipath configuration. 301 302### bdev_raid 303 304Added `bdev_raid_remove_base_bdev` RPC to remove a base bdev from existing raid bdev. 305 306### dpdk 307 308Updated DPDK submodule to DPDK 23.07. 309 310### env 311 312The `phys_addr` parameter in `spdk_*_malloc()` functions is now invalid. Passing non-NULL value 313will return NULL from the functions. The parameter was deprecated in SPDK 19.04. 314For retrieving physical addresses, `spdk_vtophys()` should be used instead. 315 316### go 317 318Added JSON-RPC 2.0 client written in Go to ease communication with SPDK. 319Please see the documentation at `go/rpc/` and example at `examples/go/`. 320 321### init 322 323Options for the JSON-RPC server initialization were added. The options are defined via the 324`spdk_rpc_opts` structure and is passed to the existing API `spdk_rpc_initialize()` as a new 325argument. The options include `log_file` and `log_level`. 326 327### intel-ipsec-mb 328 329Updated intel-ipsec-mb submodule to v1.4. 330 331### jsonrpc 332 333New APIs, `spdk_jsonrpc_set_log_level` and `spdk_jsonrpc_set_log_file`, were added to enable 334logging JSON RPC calls history. 335 336### log 337 338New APIs, `spdk_flog` and `spdk_vflog`, were added to write messages to the specified log file. 339 340### nvme 341 342The `spdk_nvme_accel_fn_table` interface has been extended with callbacks allowing users to chain 343multiple accel operations. Users can now send requests with an existing accel sequence by setting 344the `accel_sequence` field in `spdk_nvme_ns_cmd_ext_io_opts` if the controller supports it (i.e. 345sets the `SPDK_NVME_CTRLR_ACCEL_SEQUENCE_SUPPORTED` flag) and the user implements the necessary 346`spdk_nvme_accel_fn_table` callbacks. 347 348The NVMe bdev will now advertise support for accel sequences if the `allow_accel_sequence` flag is 349set in `bdev_nvme_set_options` and the underlying controller also supports them. 350 351The TCP transport will now calculate data digest using the accel sequence APIs if they're available. 352 353Added `spdk_nvme_qpair_is_connected()` API to check the connection status. 354 355### nvmf 356 357The `spdk_nvmf_request::data` field has been removed: instead, clients should set 358`->iov` and `->iovcnt` appropriately, as nvmf request APIs now expect any data 359buffers to be described there. `spdk_nvmf_request_get_data()` has been removed. 360 361`transport` field in `listen_addresses` of `nvmf_get_subsystems` RPC is deprecated. 362`trtype` field should be used instead. `transport` field will be removed in 24.01 release. 363 364Added `spdk_nvmf_subsystem_add_listener_ext()` API for accepting new connections on the provided 365address, now allowing to specify additional options for the listener. 366 367Deprecated `spdk_nvmf_subsytem_any_listener_allowed()` API (to be removed in 24.01) and replaced with 368spdk_nvmf_subsystem_any_listener_allowed()` API fixing the typo. 369 370Added `spdk_nvmf_subsystem_is_discovery()` API to check whether a given susbystem is discovery subsystem. 371 372### scripts 373 374`setup.sh` now supports interactive mode for device selection and hugepage reservation. 375 376Added `backport.sh` script that formalizes process of backporting commits from latest SPDK, 377to specific release branches. Please see [documentation](https://spdk.io/doc/backporting.html). 378 379### thread 380 381Added `spdk_thread_is_app_thread()` API to check if specified spdk_thread is the app thread. 382 383### util 384 385Extended DIF support to CRC-32 and CRC-64 format of the Protection Information. 386 387Added `spdk_crc64_nvme()` and `spdk_crc32c_nvme()`, CRC-64 and CRC-32C checksums compatible with 388NVMe Protection Information. 389 390## v23.05 391 392### accel 393 394Added API `spdk_accel_submit_xor` to perform XOR. 395 396Configuration of `mlx5_pci` driver of `accel_dpdk_cryptodev` module was changed. Now key and key2 397should have equal length and key size in hexlified form can be either 32 bytes for 128b XTS key or 39864 bytes for 256b XTS key. Support of the wrapped crypto credentials mode for this driver is 399dropped, only plain text mode is supported. 400 401The accel library will now collect statistics on the number of executed operations and processed 402bytes. These statistics can be retreived using the `accel_get_stats` RPC. 403 404Added support for appending a crc32 calculation to an accel sequence. 405 406### bdev 407 408A new API `spdk_bdev_module_claim_bdev_desc` was added. Unlike `spdk_bdev_module_claim_bdev`, this 409function requires a bdev descriptor to be passed and the claim is automatically released when the 410descriptor is closed. It allows bdev modules to claim bdevs as a single writer, multiple writers, or 411multiple readers. 412 413New APIs `spdk_bdev_quiesce`, `spdk_bdev_unquiesce`, `spdk_bdev_quiesce_range` and 414`spdk_bdev_unquiesce_range` were added. These allow I/O to be quiesced on an entire bdev or 415a specified LBA range. 416 417Removed `spdk_bdev_ext_io_opts` from `spdk_bdev_io` and replaced it with fields that were missing in 418`spdk_bdev_io`. 419 420It is now possible to submit an I/O with an associated accel sequence. All operations in such 421sequence will be executed prior to sending the I/O (writes) or after its completion (reads). 422 423### env 424 425New functions `spdk_env_get_main_core` and `spdk_env_get_cpuset` were added. 426 427It is now possible to run applications on cores with IDs greater than 128 by passing a core map 428using the `--lcores` parameter. 429 430### examples 431 432`examples/nvme/perf` application now accepts `--use-every-core` parameter that changes 433the existing worker and namespace association logic to access every namespace from each worker. 434This replicates behavior of bdevperf application when `-C` option is provided. 435 436### gpt 437 438GPT bdevs now use the GPT Unique Partition ID as the bdev's UUID. 439 440### idxd 441 442Removed the `SPDK_IDXD_FLAG_PERSISTENT` flag. 443 444### lvol 445 446New API `spdk_lvol_iter_immediate_clones` was added to iterate the clones of an lvol. 447 448New APIs `spdk_lvol_get_by_uuid` and `spdk_lvol_get_by_names` to get lvols by the lvol's UUID or 449lvstore and lvol names. 450 451New `bdev_lvol_get_lvols` RPC to list logical volumes. This provides information about logical 452volumes without providing information about the bdevs. It is useful for listing the lvols 453associated with specific lvol stores and for listing lvols that are degraded and have no 454associated bdev. 455 456Added support for external snapshots making it possible to create clones of any (including non-lvol) 457bdev. Users can create such clones using the `bdev_lvol_clone_bdev` RPC. 458 459### nvme 460 461New API `spdk_nvme_ns_get_format_index` was added to calculate the exact format index, that 462was used to format the namespace. 463 464Added two new APIs `spdk_nvme_ns_cmd_io_mgmt_recv` and `spdk_nvme_ns_cmd_io_mgmt_send` to 465receive and send the I/O management commands. 466 467### nvmf 468 469New `spdk_nvmf_request_copy_to/from_buf()` APIs have been added, which support 470iovecs, unlike the deprecated `spdk_nvmf_request_get_data()`. 471 472Two functions related to Asynchronous Event and error handling have been made public: 473 474- `spdk_nvmf_ctrlr_async_event_error_event`, 475- `spdk_nvmf_ctrlr_abort_aer`. 476 477New `spdk_nvmf_transport_create_async` was added, it accepts a callback and callback argument. 478`spdk_nvmf_transport_create` is marked deprecated. 479 480Parameters `cb_fn` and `ctx` of `spdk_nvmf_qpair_disconnect` API are deprecated. These parameters 481will be removed in 23.09 release. 482 483Added a `secure_channel` parameter to the `nvmf_subsystem_add_listener` RPC. When true, all 484connections established via this listener will immediately attempt to establish a secure channel, 485prior to any authentication. Only valid for the TCP transport. 486 487Added two optional transport module callbacks: `subsystem_add_host()` and `subsystem_remove_host()`. 488These functions will notify a transport about adding/removing hosts' access. 489 490TLS PSK identity is now generated from subsystem NQN and host NQN. PSK interchange format is now 491expected as input, when configuring TLS in SPDK. TLS feature is considered experimental. 492 493### part 494 495New API `spdk_bdev_part_construct_ext` is added and allows the bdev's UUID to be specified. 496 497### sock 498 499Added a zero copy receive interface allowing users to provide buffers to a socket group (via 500`spdk_sock_group_provide_buf()`), which will then get filled with the next portions of the stream on 501each socket. After that, pointer to the buffer holding the next part of the data can be obtained 502using `spdk_sock_recv_next()`. 503 504Added a callback `get_key()` to `spdk_sock_impl_opts` structure. 505 506New function `spdk_sock_get_impl_name()` was added to retrieve current socket implementation name. 507 508### thread 509 510Added two new APIs `spdk_thread_bind` and `spdk_thread_is_bound` to bind or unbind `spdk_thread` 511to its current CPU core, and check bound status. 512 513### ublk 514 515Enabled support for `UBLK_F_NEED_GET_DATA`. The buffers are no longer pre-allocated during 516initialization, but are allocated on-demand. Additionally, they're now allocated from the `iobuf` 517buffer pools. 518 519### util 520 521New APIs `spdk_uuid_is_null` and `spdk_uuid_set_null` were added to compare and 522set UUID to NULL value. 523 524Added new functions, `spdk_fd_group_{nest,unnest}`, providing efficient way of adding one 525`fd_group`'s file descriptors to another `fd_group`. 526 527## v23.01 528 529### accel 530 531New library isa-l-crypto has been added, it is used by accel library in crypto operations. 532 533New functions `spdk_accel_submit_encrypt` and `spdk_accel_submit_decrypt` were added. 534 535New accel module `dpdk_cryptodev` has been added. It uses DPDK crypto PMD and support encrypt and 536decrypt operations. New RPC `dpdk_cryptodev_scan_accel_module` has been added to enable this accel module. 537 538New accel module `mlx5` was added. It implements crypto operations, enabled when SPDK is configured with 539RDMA provider is mlx5_dv and crypto support. 540 541Introduced the concept of chaining multiple accel operations together and executing them all in a 542single call. The operation can be chained via one of the `spdk_accel_append_*` functions and then 543executed using `spdk_accel_sequence_finish`. Currently, copy, decompress, decrypt, encrypt, and 544fill operations support this mode. This feature is considered experimental. 545 546### bdev 547 548A new API `spkd_bdev_part_submit_request_ext` was added to specify a custom completion callback. 549 550New APIs `spdk_bdev_for_each_channel` and `spdk_bdev_for_each_channel_continue` and 551associated function pointers were added to iterate each channel of the required bdev. 552 553The RPC `bdev_get_iostat` now allows a user to query the per channel IO statistics for 554required bdev, and displays maximum and minimum latencies and I/O error counts. 555 556New `spdk_bdev_copy_blocks` and `spdk_bdev_get_max_copy` APIs to support copy commands. 557 558A new API `spdk_bdev_io_get_submit_tsc` was added to get the submit_tsc of the bdev I/O. 559 560Bdevs will no longer have UUIDs generated based on timestamp and are responsible for 561setting this field themselves. Generation of UUIDs for NVMe bdevs may be enabled by 562running `bdev_nvme_set_options` RPC with `--generate-uuids` option. These identifiers 563are based on serial number and namespace ID and will always be the same for a given 564device. 565 566A new API `spdk_bdev_channel_get_histogram` was added to get the histogram of a specified 567channel for a bdev. 568 569Converted internal use of `pthread_mutex_t` to `struct spdk_spinlock`. Consumers of bdev 570API functions must be on an SPDK thread or the program will abort. It is now enforced 571that no internal bdev locks can be held when a poller or message goes off CPU. 572 573A new RPC `bdev_reset_iostat` was added to reset I/O statistics of bdevs. Note that if one 574consumer reset I/O statistics, it affects all other consumers. 575 576Add function pointers, `dump_device_stat_json` and `reset_device_stat` to the bdev module 577function table to display and reset I/O statistics specific for the module specific bdev 578context. 579 580New APIs `spdk_bdev_reset_io_stat`, `spdk_bdev_add_io_stat` and `spdk_bdev_dump_io_stat_json` 581were added to process I/O statistics outside the generic bdev layer, especially in bdev modules. 582 583### bdev_crypto 584 585vbdev_crypto is updated to use accel framework instead of DPDK PMDs. 586 587### bdev_daos 588 589New RPC `bdev_daos_resize` was added to allow resizing the daos bdev. 590 591### bdev_malloc 592 593Both of interleaved and separated metadata are now supported by the malloc bdev module. 594 595Protection information is now supported by the malloc bdev module. 596 597### bdev_nvme 598 599Updated `bdev_nvme_set_options` RPC (and rpc.py) to support the new `transport_tos` parameter. 600 601For the active-active policy of the multipath mode, in addition to the default round-robin path 602selector, the minimum queue depth path selector was added. The minimum queue depth path selector 603selects an I/O path according to the number of outstanding requests of each nvme qpair. 604 605Added RPCs `bdev_nvme_start_mdns_discovery`, `bdev_nvme_get_mdns_discovery_info` and 606`bdev_nvme_stop_mdns_discovery` to perform Avahi based mDNS discovery service, 607as per NVMe TP 8009 - Automated Discovery of Ethernet Discovery Controllers. 608 609A new option `nvme_error_stat` was added to the `bdev_nvme_set_options` RPC to enable 610collecting NVMe error counts. 611 612Added I/O statistics per I/O path to the NVMe bdev module for NVMe bdev multipath. It can be 613enabled by a new option `io_path_stat` of RPC `bdev_nvme_set_options`. 614 615Added RPC `bdev_nvme_get_path_iostat` to get I/O statistics for IO paths of the NVMe bdev. 616 617Added `selector` parameter to `bdev_nvme_set_multipath_policy` RPC to set path selector for multipath. 618Option `round_robin` and `queue_depth` are available. 619 620Added `rr_min_io` option to RPC `bdev_nvme_set_multipath_policy`. It switches I/O to 621another path after rr_min_io I/Os are routed to current io path for the round-robin 622path selector. 623 624Added option `--io-path-stat` for RPC `bdev_nvme_set_option` to enable collecting io path stat. 625 626### bdev_raid 627 628Changed `bdev_raid_get_bdevs` RPC output format to include raid_bdev details. 629 630### bdevperf 631 632Promoted the application to example to match similar programs: fio_plugin and perf. 633It can now be found inside `examples/bdev/bdevperf`. 634 635### env 636 637Added `spdk_mempool_mem_iter` that allows to get start address and length of each memory 638chunk in order to create app-specific resources. 639 640### event 641 642Added core lock file mechanism to prevent the same CPU cores from being used by multiple 643SPDK application instances. 644 645Added `--disable-cpumask-locks` command line switch to disable CPU locks on SPDK startup. 646 647Added RPCs `framework_enable_cpumask_locks` and `framework_disable_cpumask_locks` to enable 648and disable CPU core locks in runtime. 649 650Added `--rpcs-allowed` command line option. Users can specify a comma-separated list of RPC 651names with this option to restrict allowed RPCs to only that list. 652 653Added `--vfio-vf-token` command line option to specify a VF token (UUID) 654shared between SR-IOV PF and VFs for vfio_pci driver. 655 656The `spdk_iov_xfer_*()` APIs were added for handling incremental copying between 657an iovec and a buffer. 658 659### json 660 661Added API `spdk_json_write_double` and `spdk_json_write_named_double` to allow 662for writing and decoding of the the double data type. 663 664### log 665 666Added consistent tracking of use of deprecated behavior. Deprecations continue to be 667listed in deprecation.md and now are in subsections with headers matching a tag 668used in `SPDK_LOG_DEPRECATED()` calls. When deprecated behavior is used, these 669tags will appear in SPDK's log at the warn level. As the SPDK application exits, it 670will log a summary of how many times `SPDK_LOG_DEPRECATED()` was called for each 671tag that was logged at least once. 672 673### nvme 674 675NVMe transport options were newly introduced. The NVMe transport options are defined via 676the `spdk_nvme_transport_opts` structure and configured via `spdk_nvme_transport_get_opts` 677and `spdk_nvme_transport_get_opts` functions. 678 679Shared receive queue was supported by the RDMA transport. It can be configured by 680a new NVMe transport option `rdma_srq_size`. 681 682Added `transport_tos` to `spdk_nvme_ctrlr_opts` to support setting of the "type of service" 683value in the IPv4 header. Only RDMA is supported at this time. 684 685Added API `spdk_nvme_qpair_get_num_outstanding_reqs` to get the number of outstanding reqs of 686a specified qpair. 687 688### nvmf 689 690Added API `spdk_nvmf_tgt_pause_polling` and `spdk_nvmf_tgt_resume_polling` to allow 691pausing polling on poll group of a given target. 692 693The `num_shared_buffers` option now controls the maximum number of buffers a transport will 694pull from the central `iobuf` pool. It will not allocate additional memory. If the requested 695amount cannot be satisfied, a warning will be displayed to increase the size of the `iobuf` 696pool. 697 698### rpc 699 700Added `spdk_rpc_set_allowlist` to restrict allowed RPCs to the specified list. 701 702### scheduler 703 704Changing scheduler from dynamic back to static is no longer possible, 705as there is no way of saving original SPDK thread distribution on reactors. 706 707### thread 708 709Added `spdk_thread_get_app_thread` which returns the first thread that was created using 710`spdk_thread_create`. 711 712Added `spdk_thread_is_running`. This returns `true` for a running thread, or `false` if 713its exit process has started using `spdk_thread_exit`. 714 715Added API `spdk_spin_init`, `spdk_spin_destroy`, `spdk_spin_lock`, `spdk_spin_unlock`, and 716`spdk_spin_held` to support spinlocks that are aware of the SPDK concurrency model. 717 718Added iobuf buffer pool, set of API calls for allocating data buffers across libraries. 719Please see new API `spdk_iobuf_*`. 720 721### trace 722 723New `trace_get_info` RPC was added to get name of shared memory file, list of the 724available trace point groups and mask of the available trace points for each group. 725 726### ublk device 727 728The ublk application supports the ublk kernel driver. It's implemented as a ublk backend 729in spdk_tgt and could be started with specifying configuration. See the 730[ublk](https://www.kernel.org/doc/html/latest/block/ublk.html) documentation for more details. 731 732ublk bdev could export a block device via Linux ublk. It will move this backend device into userspace 733as `/dev/ublkb*`. Before to adding ublk device, need to create ublk target by RPC method. 734 735### util 736 737New API `spdk_fd_group_get_epoll_event` that returns the epoll(7) event that 738caused a function callback in file descriptor group to execute. 739 740Added API `spdk_strarray_from_string`, `spdk_strarray_dup` and `spdk_strarray_free` 741for handling arrays of strings. 742 743A new API `spdk_strcpy_replace` was added to replace all occurrences of the search string 744with the replacement string. 745 746New API `spdk_iov_memset()` was added to memset an iovec. 747 748New API `spdk_iov_one()` was added to initialize an iovec for a single buffer. 749 750## v22.09 751 752### accel 753 754Many names were changed in the accel framework to make them consistent both with themselves and 755the rest of SPDK. The primary public header file is now named `include/spdk/accel.h`. 756 757Added a new runtime RPC `accel_get_opc_assignments` to get a list of current opcode to engine 758assignments. 759 760Added a new startup RPC `accel_assign_opc` to assign/override a specific opcode to 761an engine. 762 763### bdev 764 765New RPCs `bdev_xnvme_create` and `bdev_xnvme_delete` were added to support the xNVMe bdev. 766 767A new API `spdk_bdev_for_each_bdev_io` was added to execute the function on the appropriate 768thread for each bdev_io submitted to the bdev. 769 770A new API `spdk_bdev_get_current_qd` was added to measure and return the queue depth from a 771bdev. This API is available even when queue depth sampling is disabled. 772 773Added `spdk_bdev_seek_data`, `spdk_bdev_seek_hole` and `spdk_bdev_io_get_seek_offset` functions 774that start from a given offset and seek for next data or for next hole. At this time only 775supported by lvol bdev module. 776 777A new parameter `io_drain_timeout` has been added to `spdk_bdev` structure. It controls how long 778a bdev reset must wait for IO to complete prior to issuing a reset to the underlying device. 779If there is no outstanding IO at the end of that period, the reset is skipped. 780Best defined for bdevs that share an underlying bdev, such as multiple lvol bdevs sharing 781an nvme device, to avoid unnecessarily resetting the underlying bdev and affecting other 782bdevs that are sharing it. 783Modifying this field is optional. Setting the value to 0 keeps the original behavior 784to always send resets immediately, even if there is no I/O outstanding. 785 786### blobstore 787 788Reserve space for `used_clusters` bitmap. The reserved space could be used for blobstore growing 789in the future. 790 791Added `is_zeroes` operation to `spdk_bs_dev`. It allows to detect if logical blocks are backed 792by zeroes device and do a shortcut in copy-on-write flow by excluding copy part from zeroes device. 793 794Added `spdk_blob_get_next_allocated_io_unit` and `spdk_blob_get_next_unallocated_io_unit` functions 795that start from a given offset and seek for first io_unit belonging to an allocated cluster 796or first io_unit belonging to an unallocated cluster. 797 798Added `spdk_bs_grow` function to grow blobstore size if the underlying bdev size is increased. 799This is used by lvol bdev via `bdev_lvol_grow_lvstore` RPC. 800 801### daos 802 803Added new DAOS bdev module, that creates SPDK block device on top of DAOS DFS. 804Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_daos) for more details. 805 806### env 807 808Added `spdk_pci_register_device_provider` and matching `SPDK_PCI_REGISTER_DEVICE_PROVIDER` 809functions for registering PCI device providers. That allow the VMD driver to be notified 810when users want to attach a device under a given BDF and when a device is detached. 811 812### init 813 814`spdk_subsystem_init_from_json_config` now fails if the JSON configuration file is not 815an object with an array named "subsystems". 816 817### iscsi 818 819Added `bdev_iscsi_set_options` RPC to modify options of the iSCSI bdev module. 820 821### json 822 823Added `spdk_json_find` API return errcode: EPROTOTYPE - json not enclosed in {}. 824`spdk_json_find` now returns -EPROTOTYPE instead of -ENOENT if the object parameter 825does not point to a JSON object (i.e. is not enclosed with {}). 826 827### lvol 828 829Add `num_md_pages_per_cluster_ratio` parameter to the `bdev_lvol_create_lvstore` RPC. 830Calculate `num_md_pages` from `num_md_pages_per_cluster_ratio`, and pass it to `spdk_bs_opts`. 831 832### nvme 833 834Added SPDK_NVME_TRANSPORT_CUSTOM_FABRICS to enum `spdk_nvme_transport_type` to support custom 835fabric transport. SPDK_NVME_TRANSPORT_CUSTOM was intended to be non-fabric custom transport. 836 837Added a new function `spdk_nvme_ns_cmd_verify` to submit a Verify Command to a Namespace. 838 839Added `spdk_nvme_ctrlr_disable_read_changed_ns_list_log_page` to allow an application to 840tell the driver to not read the CHANGED_NS_LIST log page in response to a NS_ATTR_CHANGED 841AEN. When called the application is required to read this log page instead to clear the 842AEN. 843 844Added `psk` field to `spdk_nvme_ctrlr_opts` struct in order to enable SSL socket implementation 845of TCP connection and set the PSK. Applicable for TCP transport only. 846 847### raid 848 849Renamed the `raid5` module to `raid5f` to reflect that it is not a traditional 850RAID5 implementation - only full stripe writes are supported, partial stripe 851writes (read-modify-write) are not. 852 853### rpc 854 855Added `psk` parameter to `bdev_nvme_attach_controller` RPC in order to enable SSL socket implementation 856of TCP connection and set the PSK. Applicable for TCP transport only. 857 858New options `enable_ktls` and `tls_version` were added to the `sock_impl_set_options` structure. 859New options `psk_key` and `psk_identity` were added to the `sock_impl_set_options` structure. 860 861Added warning message for `bdev_rbd_create`, if it is used without -c. 862`bdev_rbd_create()` API without specifying -c is deprecated and will be removed in future release. 863 864Renamed `enable_vmd` RPC to `vmd_enable` to make it consistent with our naming scheme of 865`<subsystem>_<action>`. For now, the old name is still available, but is marked as deprecated. 866 867New function `spdk_rpc_get_method_state_mask` was added to RPC library. 868 869### sma 870 871Extended `VolumeParameters` with crypto parameters allowing the user to configure crypto when 872attaching a volume to a device. This interface is now supported by all upstream device types 873(nvmf-tcp, virtio-blk, vfio-user) using `bdev_crypto`. Users must specify the crypto engine to use 874under `crypto` section in config. It is also possible to register out-of-tree crypto engines by 875inheriting from the `CryptoEngine` class. 876 877Added two new methods: `SetQos` and `GetQosCapabilities` allowing the user to configure QoS on a 878per-device or per-volume level. Not all QoS settings have to be supported by each device, so users 879can use `GetQosCapabilities` to query them for that. All upstream device types support QoS on a 880per-volume level using bdev layer's QoS mechanism. 881 882### sock 883 884Added new `ssl` based socket implementation, the code is located in `module/sock/posix`. 885For now we are using hard-coded PSK and only support TLS 1.3. 886 887### util 888 889Added new functions: `spdk_hexlify` and `spdk_unhexlify`. 890 891A new API `spdk_xor_gen` was added to generate XOR from multiple source buffers. It is going to be 892used by `raid5f` for calculating parity. 893 894### vfu_tgt 895 896Added vfu_tgt library abstraction based on libvfio-user for PCI device emulation. 897Besides the NVMe device emulation, its now possible to emulate virtio-blk and virtio-scsi 898devices. 899 900### virtio 901 902virtio-vhost-user no longer tries to support dynamic memory allocation. The vhost target does 903not support the high rate of SET_MEM_TABLE vhost messages that result from dynamic memory 904allocation, so a virtio-vhost-user device will now present an ERRLOG, assert, and skip the 905SET_MEM_TABLE vhost message if a memory notification is received outside of the normal device 906start/stop. Applications using the virtio library in vhost-user mode should now pre-allocate 907the application's memory using the -s/--mem-size option and use single shared memory file 908segments using the -g/--single-file-segments option. 909 910Added `vfio-user` transport type for virtio-blk and virtio-scsi devices. 911 912### vmd 913 914Fixed hotplug when a device is inserted in a slot in which a disk was already enumerated previously. 915Added two new RPCs: `vmd_remove_device` simulating a hotremove, and `vmd_rescan`, which rescans all 916buses managed by the VMD driver and hotplugs all newfound devices. 917 918## v22.05 919 920### sock 921 922A new option `ack_timeout` was added to the `spdk_sock_opts` structure. 923 924### util 925 926A new parameter `bounce_iovcnt` was added to `spdk_dif_generate_copy` and `spdk_dif_verify_copy`. 927The `bounce_iovcnt` is used to specify the number of bounce_iov to support multiple block-aligned 928fragment copies. 929 930A new API `spdk_copy_iovs_to_buf` and `spdk_copy_buf_to_iovs` were added to copy iovs to buf or 931copy buf to iovs. There're many cases need to use these two APIs. 932 933### bdev 934 935Removed deprecated spdk_bdev_module_finish_done(). Use spdk_bdev_module_fini_done() instead. 936 937A new API `spdk_bdev_unregister_by_name` was added to handle race conditions correctly. 938 939New APIs, `spdk_for_each_bdev` and `spdk_for_each_bdev_leaf`, were added to provide iteration 940safe for race conditions. 941 942A new RPC `bdev_nvme_get_io_paths` was added to get all active I/O paths. 943 944A new RPC `bdev_nvme_set_preferred_path` was added to set preferred I/O path for an NVMe bdev 945when in multipath mode. This RPC does not support NVMe bdevs in failover mode. 946 947A new RPC `bdev_nvme_set_multipath_policy` was added to set multipath policy of a NVMe bdev 948in multipath mode. 949 950A new option `disable_auto_failback` was added to the `bdev_nvme_set_options` RPC to disable 951automatic failback. 952 953### idxd / dsa 954 955A new parameter `flags` was added to all low level submission and preparation 956APIs to enable the caller to pass operation flags per the DSA specification. 957 958A new flag 'SPDK_IDXD_FLAG_PERSISTENT' was added to let DSA know that 959the destination is persistent. 960 961The RPC `idxd_scan_accel_engine` has been renamed to `dsa_scan_accel_engine` 962 963The RPC `iaa_scan_accel_engine` has been added. 964 965Many HW related structs/functions with the name `idxd` have been renamed `dsa` 966to more accurately represent the HW they are associated with. 967 968Two new functions were added to the library `spdk_idxd_submit_compress` and 969`spdk_idxd_submit_decompress` 970 971### accel_fw 972 973A new parameter `flags` was added to accel API. 974The APIs include: 975`spdk_accel_submit_copy` 976`spdk_accel_submit_dualcast` 977`spdk_accel_submit_fill` 978`spdk_accel_submit_copy_crc32c` 979`spdk_accel_submit_copy_crc32cv` 980`spdk_accel_submit_compress` 981`spdk_accel_submit_decompress` 982 983A new flag `ACCEL_FLAG_PERSISTENT` was added to indicate the target memory is PMEM. 984 985The API `spdk_accel_get_capabilities` has been removed. 986 987### crypto 988 989Support for AES_XTS was added for MLX5 polled mode driver (pmd). 990 991bdev_crypto_create RPC now requires hexlified 'key' and 'key2' params for all pmd drivers. 992Unhexlifying is performed during RPC command processing and the vbdev crypto module runs on 993binary keys as before. 994 995### bdev 996 997Added a timeout option to the `bdev_get_bdevs` RPC. It allows the user to specify the amount of 998time to wait until a bdev with a given name appears in the system. 999 1000### bdev_nvme 1001 1002Added `bdev_nvme_add_error_injection` and `bdev_nvme_remove_error_injection` RPCs to add and 1003remove NVMe error injections. 1004 1005New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 1006added to the RPC `bdev_nvme_set_options`. They can be overridden if they are given by the RPC 1007`bdev_nvme_attach_controller`. 1008 1009### blobstore 1010 1011New functions `spdk_blob_io_writev_ext` and `spdk_blob_io_readv_ext` are added. The new functions accept 1012`spdk_blob_ext_io_opts` structure with extended IO request options. 1013 1014### event 1015 1016Added `msg_mempool_size` parameter to `spdk_reactors_init` and `spdk_thread_lib_init_ext`. 1017The size of `g_spdk_msg_mempool` can now be controlled through the same-named 1018user option of `spdk_app_opts` structure. 1019 1020### nvme 1021 1022The API `spdk_nvme_ctrlr_prepare_for_reset()` was deprecated. The functionality provided by the 1023`spdk_nvme_ctrlr_prepare_for_reset()` was merged into the API `spdk_nvme_ctrlr_disconnect()`. 1024 1025### nvmf 1026 1027Removed deprecated max_qpairs_per_ctrlr parameter from nvmf_create_transport RPC. Use 1028max_io_qpairs_per_ctrlr instead. 1029 1030Deprecated the ability for hosts to connect to the discovery subsystem automatically on any 1031existing listener. Users should now explicitly add listeners for the discovery subsystem. 1032Host can still connect to the discovery subsystem as before, but a warning message will be 1033emitted if no listener was configured for the transport ID of the incoming connection. 1034 1035Added adaptive interrupt feature for vfio-user transport. New parameter `disable_adaptive_irq` 1036is added to the RPC `nvmf_create_transport`. 1037 1038### thread 1039 1040Added `spdk_thread_exec_msg()` API. 1041 1042### scheduler 1043 1044`framework_set_scheduler` can now be called after application initialization. 1045Added callbacks to set custom parameters specific for each scheduler implementation 1046and `framework_get_scheduler` to retrieve them. 1047 1048Added `dynamic` scheduler options: load_limit, core_limit, core_busy. Their descriptions 1049are available in JSON-RPC document, in section 1050[framework_set_scheduler](jsonrpc.html#rpc_framework_set_scheduler). 1051 1052### raid 1053 1054Add concat as a special raid module. The concat module could create a virtual bdev. The 1055virtual bdev combines multiple underlying bdevs together. The layout of the underlying 1056bdevs is one after another. The concat bdev is extendable. When the free space of the 1057concat bdev is not enough, the user can deconstruct the concat bdev, then reconstruct it 1058with an additional underlying bdev. 1059 1060### sock 1061 1062Allow MSG_ZEROCOPY flag to be set or not according to data size, which can be enabled and 1063set by setting "zerocopy_threshold". zerocopy_threshold = 0 means disable this function; 1064zerocopy_threshold > 0 means enable it and use this value as the threshold. 1065 1066### rpc 1067 1068Introduced `zerocopy_threshold` to enable zerocopy on send for server sockets according to 1069data size to be flushed. 1070 1071## v22.01 1072 1073### accel 1074 1075The batching capability was removed. Batching is now considered an implementation 1076detail of the low level drivers. 1077 1078### bdev 1079 1080The NVMe bdev module supports multipath and improved I/O error resiliency. 1081 1082The parameter `retry_count` of the RPC `bdev_nvme_set_options` was deprecated and will be 1083removed in SPDK 22.04, and the parameter `transport_retry_count` is added and used instead. 1084 1085An new parameter `bdev_retry_count` is added to the RPC `bdev_nvme_set_options`. 1086 1087New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 1088added to the RPC `bdev_nvme_attach_controller`. 1089 1090An new parameter `num_io_queues` is added to `bdev_nvme_attach_controller` RPC to allow specifying amount 1091of requested IO queues. 1092 1093Added `key_file` parameter to the `rbd_register_cluster` RPC. It is an optional parameter to 1094specify a keyring file to connect to a RADOS cluster. 1095 1096Added `spdk_bdev_get_zone_id` API to retrieve zone_id for a certain LBA. 1097 1098### bdev_aio 1099 1100Added `bdev_aio_rescan` RPC to allow rescanning the size of aio bdev. 1101 1102### bdev_nvme 1103 1104Added discovery service to bdev_nvme module, which now can connect to a discovery controller. 1105bdev_nvme will connect to all current and future subsystems in the discovery controller. 1106Added `bdev_nvme_start_discovery` and `bdev_nvme_stop_discovery` RPC. 1107 1108### dpdk 1109 1110Updated DPDK submodule to DPDK 21.11. 1111 1112### env 1113 1114Added `spdk_pci_for_each_device`. 1115 1116Removed `spdk_pci_get_first_device` and `spdk_pci_get_next_device`. These APIs were unsafe, because 1117they did not account for PCI devices being inserted or removed while the caller was using handles 1118returned from these APIs. Existing users of these APIs should switch to `spdk_pci_for_each_device` 1119instead. 1120 1121Added 3 experimental APIs to handle PCI device interrupts (`spdk_pci_device_enable_interrupt`, 1122`spdk_pci_device_disable_interrupt`, `spdk_pci_device_get_interrupt_efd`). 1123 1124Added `framework_get_pci_devices` RPC to list PCIe devices attached to an SPDK application. 1125 1126### idxd 1127 1128Many APIs are now vectored rather than scalar, meaning they take iovecs instead of individual pointers. 1129 1130### json 1131 1132Added `spdk_json_write_bytearray` API to serialize a buffer as a hex string. 1133 1134### nvme 1135 1136API `spdk_nvme_trtype_is_fabrics` was added to return existing transport type 1137is fabric or not. 1138 1139API `spdk_nvme_poll_group_remove` was limited to be available only for a 1140disconnected qpair in the group. 1141 1142New APIs, `spdk_nvme_ctrlr_disconnect`, `spdk_nvme_ctrlr_reconnect_async`, and 1143`spdk_nvme_ctrlr_reconnect_poll_async`, have been added to improve error recovery, and 1144the existing APIs,`spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async` 1145were deprecated. 1146 1147Added `spdk_nvme_ctrlr_get_discovery_log_page` API for getting the full discovery log page 1148from a discovery controller. 1149 1150### nvmf 1151 1152Added support for zero-copy operations in the NVMe-oF TCP target. It can be enabled via 1153the `zcopy` parameter when creating a transport. The zero-copy operations are only used 1154for requests not using in-capsule data. 1155 1156Added a `subsystem` parameter to `spdk_nvmf_transport_stop_listen_async`. When not NULL, 1157it will only disconnect qpairs for controllers associated with the specified subsystem. 1158 1159Removed accept poller from transport layer. Each transport can have its own policy of 1160handling new connections. To notify transport layer `spdk_nvmf_poll_group_add` and 1161`spdk_nvmf_tgt_new_qpair` can be used. 1162 1163### scsi 1164 1165Structure `spdk_scsi_lun` has been extended with new member `resizing` so that SCSI layer now reports 1166unit attention for disk resize. 1167 1168### trace 1169 1170Added `spdk_trace_create_tpoint_group_mask` to return tracepoint group mask from 1171a tracepoint group name. 1172 1173Added `trace_set_tpoint_mask` and `trace_clear_tpoint_mask` RPC to allow enabling 1174individual traces. 1175 1176### util 1177 1178Added `spdk_ioviter_first` and `spdk_ioviter_next` to iterate over two iovecs and 1179yield pointers to common length segments. 1180 1181### sock 1182 1183A new parameter `hint` is added to `spdk_sock_get_optimal_sock_group`. It allows to suggest 1184a poll group when no optimal poll group is found. 1185 1186## v21.10 1187 1188Structure `spdk_nvmf_target_opts` has been extended with new member `discovery_filter` which allows to specify 1189filtering rules applied during discovery log generation. Refer to `enum spdk_nvmf_tgt_discovery_filter` for more info. 1190 1191### bdev 1192 1193New API `spdk_bdev_get_memory_domains` has been added, it allows to get SPDK memory domains used by bdev. 1194 1195New API functions `spdk_bdev_readv_blocks_ext` and `spdk_bdev_writev_blocks_ext` have been added. 1196These functions accept `spdk_bdev_ext_io_opts` structure with extended IO request 1197options, e.g. DMA memory domain which describes data that may belong to another memory domain and 1198can't be accessed directly. 1199 1200Added `async_fini_start` to allow bdev modules to complete the `fini_start` asynchronously, 1201with new `spdk_bdev_module_fini_start_done` API. 1202 1203Deprecated `spdk_bdev_module_finish_done()` API, which will be removed in SPDK 22.01. 1204Bdev modules should use `spdk_bdev_module_fini_done()` instead. 1205 1206The `ocssd` bdev has been removed. The Open Channel specification has been largely superseded by 1207zoned namespaces, few if any devices have been brought to market, and there is little reason to 1208continue to support this. OCSSD support in the nvme driver will remain for now. 1209 1210### dma 1211 1212A new library, lib/dma, has been added. This library provides the necessary infrastructure for 1213handling systems and devices with multiple memory domains. For example, a PCIe add-in card with an 1214SoC may be running SPDK on the SoC. That SoC has its own local memory, but SPDK may be controlling 1215devices that can also access the host system memory. This library provides infrastructure to enumerate 1216the memory domains and request hardware perform DMA transfers between them. 1217 1218### dpdk 1219 1220Updated DPDK submodule to DPDK 21.08. 1221 1222### event 1223 1224Added the `disable_signal_handlers` flag to the `spdk_app_opts` struct. 1225 1226### idxd 1227 1228Added `spdk_idxd_get_socket` to query the socket that the idxd device 1229is on. 1230 1231### json 1232 1233Added API to allow for writing and decoding of new types: 1234 1235- `spdk_json_write_named_uint16` 1236- `spdk_json_write_named_uint8` 1237- `spdk_json_write_uint16` 1238- `spdk_json_write_uint8` 1239- `spdk_json_number_to_uint8` 1240- `spdk_json_decode_uint8` 1241 1242### log 1243 1244Added API `spdk_log_to_syslog_level` to return syslog level based on SPDK's 1245log level. 1246 1247### nvme 1248 1249Added new functions `spdk_nvme_zns_set_zone_desc_ext` and `spdk_nvme_zns_ext_report_zones` 1250to set zone descriptor extension and to get extended zone report respectively. 1251 1252New API `spdk_nvme_ctrlr_get_memory_domains` has been added, it allows to get SPDK memory domains used by nvme controller. 1253 1254New API functions `spdk_nvme_ns_cmd_readv_ext` and `spdk_nvme_ns_cmd_writev_ext` 1255have been added. These functions accept `spdk_nvme_ns_cmd_ext_io_opts` structure with extended IO request 1256options, e.g. DMA memory domain which describes data that may belong to another memory domain and 1257can't be accessed directly. 1258 1259Added a new function `spdk_nvme_ctrlr_get_regs_bpinfo` to get boot partition info of a controller. 1260Added new functions `spdk_nvme_ctrlr_write_boot_partition`, 1261`spdk_nvme_ctrlr_read_boot_partition_start` and `spdk_nvme_ctrlr_read_boot_partition_poll` 1262to write and read the boot partitions of a controller. 1263 1264Added `spdk_nvme_ctrlr_get_opts` to retrieve the current controller options. 1265 1266Added `async_mode` to `spdk_nvme_io_qpair_opts` to enable creation of submission and completion 1267queues asynchronously. This mode is currently supported at PCIe layer, 1268which tracks the qpair creation with state machine and returns to the user immediately. 1269Default mode is set to false to create io qpairs synchronously. 1270 1271Added `spdk_nvme_ctrlr_get_regs_cc` to retrieve NVMe controller CC (Configuration) register. 1272 1273Added `spdk_nvme_ctrlr_prepare_for_reset` to inform the driver that the application is preparing 1274to reset the specified NVMe controller. This function allows the driver to make decisions 1275knowing that a reset is about to happen. 1276 1277### nvmf 1278 1279Added `oncs` to `struct spdk_nvmf_ctrlr_data` so that the transport layer 1280can decide support RESERVATION feature or not. 1281 1282An `opts_size` element was added in the `spdk_nvmf_ns_opts` structure to solve the 1283ABI compatibility issue between different SPDK version. An new option `anagrpid` was 1284added in the `spdk_nvmf_ns_opts` structure. 1285 1286An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_add_ns`. 1287 1288An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_listener_set_ana_state`. 1289 1290`spdk_nvmf_subsystem_destroy` is now can be asynchronous, it accepts a callback and callback argument. 1291 1292A new parameter, `poll_groups_mask` was added to the `nvmf_set_config` RPC that allows specifying 1293a subset of cores for the nvmf poll groups. This helps to avoid imbalances when some cores are 1294busy with periodic timer tasks that run very frequently. 1295 1296A new parameter, `commit` was added to `nvmf_bdev_ctrlr_end_zcopy` that allows indicating whether 1297the buffers should be committed. 1298 1299Added `listen_dump_opts` to `spdk_nvmf_transport_ops` so that the transport layer can display 1300listen opts. 1301 1302Added `fabrics_connect_timeout_us` to `bdev_nvme_attach_controller` RPC to allow specifying 1303timeout for connect operation. 1304 1305Added `multipath` to `bdev_nvme_attach_controller` RPC to allow specifying multipath behavior. 1306 1307### scheduler 1308 1309New API for implementing schedulers and governors. Please see `include/spdk/scheduler.h` for details. 1310 1311### scsi 1312 1313New functions, `spdk_scsi_dev_get_first_lun` and `spdk_scsi_dev_get_next_lun` 1314have been added to iterate LUNs of a SCSI device. 1315 1316Each SCSI device supports 256 LUNs at the maximum now and the macro constant 1317`SPDK_SCSI_DEV_MAX_LUN` was removed. 1318 1319### trace_parser 1320 1321A new library, lib/trace_parser, has been added. This library provides functions that parse traces 1322recorded by an SPDK application. That includes merging traces from multiple cores, 1323sorting them by their timestamp and constructing trace entries spanning across multiple buffers. 1324 1325### util 1326 1327The `spdk_fd_group_add` API now takes a `name` parameter. 1328 1329## v21.07 1330 1331### accel_fw 1332 1333Added API `spdk_accel_submit_copy_crc32c` to perform a CRC32C while copying data. 1334 1335Added API `spdk_accel_batch_prep_copy_crc32c` to batch CRC32C + copy commands. 1336 1337Added API `spdk_accel_submit_copy_crc32cv` to submit chained CRC32C + copy commands. 1338 1339### bdev 1340 1341Change `spdk_bdev_read_blocks_with_md` arg offset definition from int64_t to uint64_t. 1342 1343Red-black tree has been used to organize the bdev names and aliases uniformly 1344to provide faster lookup. 1345 1346Removed ZCOPY emulation: The bdev module can be checked to see if it supports ZCOPY 1347and if not supported then use existing READ/WRITE commands. 1348 1349Added iov to spdk_bdev_zcopy_start. 1350 1351### dpdk 1352 1353Updated DPDK submodule to DPDK 21.05. 1354 1355### idxd 1356 1357Remove the `probe_cb` parameter in `spdk_idxd_probe` function. And remove the definition 1358of `spdk_idxd_probe_cb` function pointer. It should be implemented in idxd_user.c. 1359 1360Added API `spdk_idxd_submit_copy_crc32c` to perform a CRC32C while copying data. 1361 1362Added API `spdk_idxd_batch_prep_copy_crc32c` to prepare a batch operation to perform 1363a CRC32C while copying data. 1364 1365We have userspace idxd driver under accel_engine library (module/accel/idxd/accel_engine). 1366And kernel idxd driver usage is also added under the accel_engine library. This approach 1367implemented a wrapper library to use IDXD device by leveraging the kernel DSA driver in 1368SPDK idxd library (lib/idxd). Then users can leverage the RPC later to configure how to 1369use the DSA device by user space driver or kernel driver. 1370 1371### init 1372 1373Added new `init` library that initializes the SPDK subsystems, which previously was 1374internal to application framework. That functionality has been made public and for 1375use by applications which don't leverage SPDK's application framework. 1376 1377### iscsi 1378 1379New parameters, `pdu_pool_size`, `immediate_data_pool_size`, and `data_out_pool_size`, 1380were added to the RPC `iscsi_set_options` to run iSCSI target with varying amount of 1381available memory. 1382 1383### json 1384 1385Added API `spdk_json_write_named_uint128` and `spdk_json_write_uint128` to perform 1386the uint128 related data. 1387 1388### net 1389 1390Removed deprecated `net` library. 1391 1392Added support for ZCOPY. ZCOPY is used in preference to READ and WRITE if the 1393bdev module supports ZCOPY and the bdev module has ZCOPY enabled. 1394 1395### nvme 1396 1397`spdk_nvme_map_prps` and `spdk_nvme_map_cmd` were moved to nvmf/vfio-user as internal APIs 1398as vfio-user is the only user for the above two APIs. 1399 1400Added a new function `spdk_nvme_ns_cmd_copy` to submit a Simple Copy Command to a Namespace. 1401 1402Update the `spdk_nvme_generic_command_status_code` structure with new status code 1403according to the definition in NVMe 1.4 spec. 1404 1405`spdk_nvme_ctrlr_get_default_ctrlr_opts` now sets `use_cmb_sqs` to false. This means 1406that if a controller has a CMB and supports SQs in the CMB, SPDK will not use 1407the CMB for SQs by default - the user must set `use_cmb_sqs` to true in 1408the `spdk_nvme_ctrlr_opts` structure prior to controller attach. 1409 1410Add a new function `spdk_nvme_detach_poll` to simplify a common use case to continue 1411polling until all detachments complete. 1412 1413Added new argument `timeout_admin_us` to `spdk_nvme_ctrlr_register_timeout_callback` so callers 1414can specify a different timeout for admin commands vs. io commands. 1415 1416An existing function `spdk_nvme_detach_async` was updated to add one or more detachments 1417to an active context while it is being polled. 1418 1419Rename a variable in the member `cmic` of the struct `spdk_nvme_ctrlr_data` from 1420`multi_host` to `multi_ctrlr`. The variable means that the NVM subsystem may have two or 1421more controllers if set to 1. However `multi_host` had indicated a particular use case 1422such that the NVM subsystem is used by multiple hosts. 1423 1424A new option `disable_read_ana_log_page` was added to struct `spdk_nvme_ctrlr_opts` to disable 1425reading ANA log page. The upper layer is expected to read ANA log page instead if `true`. 1426The default value is `false`. 1427 1428New APIs, `spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async`, have been added to 1429reset a controller asynchronously. 1430 1431New RPC `bdev_nvme_reset_controller` was added, to reset an NVMe controller. 1432 1433Added `spdk_nvme_ns_get_nguid` function to get NGUID for the given namespace. 1434 1435Added `spdk_nvme_ctrlr_is_fabrics` function to indicate whether a ctrlr handle 1436is associated with a fabrics controller. 1437 1438### nvmf 1439 1440Added `min_cntlid` and `max_cntlid` to `nvmf_create_subsystem` to limit the controller ID range. 1441Added `spdk_nvmf_subsystem_get_min_cntlid` and `spdk_nvmf_subsystem_get_max_cntlid` to request those values. 1442 1443`spdk_nvmf_request_get_buffers_multi` API is removed. 1444 1445Added the `nvmf_set_crdt` RPC for setting command retry delay times. 1446 1447Expanded `spdk_nvmf_poll_group_stat` with current qpair count statistics. 1448 1449Removed following deprecated APIs: 1450 1451- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 1452- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 1453- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 1454- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 1455- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 1456- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 1457- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 1458 1459### rpc 1460 1461New RPC `bdev_rbd_register_cluster` and `bdev_rbd_unregister_cluster` was added, it allows to create 1462and delete the rados object cluster, then users can choose the cluster to create related rbd 1463device. 1464 1465Revised `bdev_rbd_create` parameter, it allows to use an optional parameter `--cluster-name` 1466to create a rbd bdev with an already registered Rados Cluster Object. 1467 1468New RPC `bdev_rbd_get_clusters_info` was added, it allows to get the info of the registered 1469Rados Cluster names. 1470 1471New optional parameter, `timeout_admin_us`, added to the bdev_nvme_set_options RPC. 1472 1473Revised a parameter `--stripe-size_kb` to `--stripe-size-kb` of `bdev_raid_create` method 1474provided in `scripts/rpc.py` for consistency. 1475 1476An new optional parameter `config_kernel_mode` was added to the RPC `idxd_scan_accel_engine`, 1477and this is used to enable using the kernel mode IDXD driver. 1478 1479Removed deprecated parameter `enable-zerocopy-send` of RPC `sock_impl_set_options`, 1480use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 1481Removed deprecated parameter `disable-zerocopy-send` of RPC `sock_impl_set_options`, 1482use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 1483 1484### rpm 1485 1486Removed deprecated `pkg/spdk.spec`, please use `rpmbuild/spdk.spec` instead. 1487See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 1488 1489### thread 1490 1491Red-black tree has been used for timed pollers to provide faster insertion and deletion 1492and for io_devices to provide faster lookup. 1493 1494Removed `spdk_io_channel` structure from public header and moved it to thread_internal.h. 1495 1496### trace 1497 1498Added `spdk_trace_register_description_ext` function to register variable number of tracepoint arguments. 1499 1500Added ability to chain multiple trace entries together to extend the size of the argument buffer. 1501 1502### util 1503 1504Red-black tree macros has been added by using the macros provided by the FreeBSD operating system 1505under the same BSD license. 1506 1507Add an new macro `SPDK_SIZEOF_MEMBER` to get the size of a member of a struct. 1508 1509`spdk_crc32c_iov_update` function was added to support calculating the crc32c of the iovs. 1510 1511Added zipf random number generator with power law probability distribution. 1512When applied to performance testing of block devices, it will select blocks over 1513the full range of LBAs, but will more frequently select lower-numbered LBAs. 1514 1515## v21.04 1516 1517### accel 1518 1519Two new accelerated crc32 functions `spdk_accel_submit_crc32cv` and 1520`spdk_accel_batch_prep_crc32cv` are added in order to provide the 1521chained accelerated CRC32 computation support. 1522 1523### bdev 1524 1525For `bdev_ocssd_create` RPC, the optional parameter `range` was removed. 1526Only one OCSSD bdev can be created for one OCSSD namespace. 1527 1528Removed the `spdk_bdev_open` from bdev library API. 1529Removed the `spdk_vbdev_register` and `spdk_bdev_part_base_construct` from bdev module API. 1530Removed the `config_text` function for bdev modules to report legacy config. 1531 1532Added `spdk_bdev_get_max_active_zones` API to display maximum number active zones of a given bdev. 1533 1534Added `spdk_bdev_get_max_zone_append_size` API to display maximum zone append data transfer size. 1535 1536### bdev_nvme 1537 1538Added support for zoned namespaces. 1539 1540### blobstore 1541 1542Removed the `spdk_bdev_create_bs_dev_from_desc` and `spdk_bdev_create_bs_dev` API. 1543 1544### env 1545 1546Added `spdk_pci_device_allow` API to allow applications to add PCI addresses to 1547the allowed list after the application has started. 1548 1549Removed the `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts`. 1550 1551Added hotplug support based on uevent in `pci_event.c`. Added usage of this functionality in 1552nvme, virtio-scsi and virtio_blk libraries. Please see the new API `spdk_pci_event_listen`, 1553`spdk_pci_get_event`, `spdk_pci_register_error_handler` and `spdk_pci_unregister_error_handler`. 1554 1555### event 1556 1557Removed the `config_file`, `max_delay_us`, `pci_whitelist` 1558and `pci_blacklist` members of struct `spdk_app_opts`. 1559 1560### idxd 1561 1562A new API `spdk_idxd_device_needs_rebalance` was added so that users of the library 1563can know whether they need to rebalance the flow control for the channel 1564that was just added/removed. This is based on how the low level library 1565shares devices amongst channels. 1566 1567The API `spdk_idxd_reconfigure_chan` had the `num_channels` removed as this 1568is now tracked in the library. The app makes use the new API above to 1569determine whether to rebalance or not. This applies to `spdk_idxd_configure_chan` 1570as well. 1571 1572The API `spdk_idxd_put_channel` now returns the rebalance state for the 1573underlying device. 1574 1575### iscsi 1576 1577A security vulnerability has been identified and fixed in the SPDK iSCSI target. 1578A TEXT PDU with no data, but CONTINUE flag set, would result in a NULL pointer dereference 1579and crash the SPDK iSCSI target process. All users of the SPDK iSCSI target 1580are recommended to update. All SPDK versions <= v21.01 are affected. 1581 1582### net 1583 1584The net library is deprecated and will be removed in the 21.07 release. 1585 1586### nvme 1587 1588Added a new function `spdk_nvme_ctrlr_get_regs_pmrcap` to get the PMR capabilities. 1589 1590Added an accelerated table pointer in `spdk_nvme_poll_group` 1591which can be used to provide the accelerated functions by users with 1592hardware engine, such as crc32c accelerated function. 1593 1594Added new functions `spdk_nvme_ctrlr_get_pmrsz`, `spdk_nvme_ctrlr_enable_pmr`, 1595`spdk_nvme_ctrlr_disable_pmr`, `spdk_nvme_ctrlr_map_pmr` and `spdk_nvme_ctrlr_unmap_pmr`. 1596 1597Added NVMe transport operations to enable, disable, map and unmap the PMR. 1598 1599Added `spdk_nvme_qpair_get_optimal_poll_group` function and `qpair_get_optimal_poll_group` 1600function pointer in spdk_nvmf_transport_ops structure in order to add the qpair to the most 1601suitable polling group. 1602 1603Added OPTPERF and namespace optimal performance fields to nvme_spec.h. 1604 1605Added `spdk_nvme_set_hotplug_filter` API to allow applications to choose which 1606hot-inserted SSDs should be probed. This is useful for use cases where multiple 1607independent SPDK processes are running on one node. The filter function can 1608then be implemented in these processes to decide which SSDs to probe based on 1609the new SSD's PCI address. 1610 1611New functions `spdk_nvme_poll_group_get_stats` and `spdk_nvme_poll_group_free_stats` 1612were added. These functions allow to get transport statistics per NVME poll group. 1613 1614Added `spdk_nvme_map_cmd` API to map the NVMe command with SGL cases. 1615 1616Added support for vector variant of ZNS zone append commands with new API 1617`spdk_nvme_zns_zone_appendv` and `spdk_nvme_zns_zone_appendv_with_md`. 1618 1619Added `spdk_nvme_zns_ns_get_max_open_zones` and `spdk_nvme_zns_ns_get_max_active_zones` API, 1620to display maximum number of open and active zones of the given namespace. 1621 1622Added `spdk_nvme_zns_ns_get_zone_size_sectors` API to provide size of zone in number of 1623sectors. 1624 1625Added `spdk_nvme_qpair_get_id` API to display the ID of the specified qpair. 1626 1627### nvmf 1628 1629Removed the `spdk_nvmf_tgt_listen` and `spdk_nvmf_subsystem_add_ns` API. 1630 1631Added new APIs: 1632 1633- `spdk_nvmf_poll_group_dump_stat` (function in `nvmf.h`). 1634- `poll_group_dump_stat` (transport op in `nvmf_transport.h`). 1635 1636The following APIs have been deprecated and will be removed in SPDK 21.07: 1637 1638- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 1639- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 1640- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 1641- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 1642- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 1643- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 1644- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 1645 1646See header files for details. 1647 1648The `trtype` field in JSON returned by `nvmf_get_stats` RPC contains now the name of the transport, 1649which is the same as the type for defined transports and more informative for a custom transport. 1650 1651Added `hdgst` and `ddgst` parameters to `bdev_nvme_attach_controller` RPC in order change 1652state of TCP header and data digest. 1653 1654Added `num_cqe` parameter to `nvmf_create_transport` RPC to set number of completion queues (CQ) 1655for RDMA transport. Useful when CQ resize operation is not supported, for example iWARP. 1656 1657### ocf 1658 1659Updated OCF submodule to v20.12.2 1660 1661Added `bdev_ocf_set_cache_mode` RPC to dynamically switch cache mode of OCF bdev. 1662 1663OCF deprecation notice has removed as Huawei is picking up support for the OCF project. 1664 1665### opal 1666 1667Removed the `spdk_opal_supported` API. 1668 1669### raid 1670 1671For `bdev_raid_create` RPC, the deprecated parameter `strip_size` was removed. 1672 1673### rpc 1674 1675New RPC `bdev_nvme_get_transport_statistics` was added, it allows to get transport statistics 1676of nvme poll groups. 1677 1678Parameter `enable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1679use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 1680Parameter `disable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1681use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 1682 1683Added cmd_parser.py used to parse the args from argparse. There are 1684two benefit to use command parser: 1685 1686- Simplify the definition of rpc method. It will reduce the rpc method code. 1687- Make the rpc call more versatile. User can add private args into rpc method. 1688 1689### rpm 1690 1691Added support for new RPM spec, rpmbuild/spdk.spec, which can be used for packaging the 1692SPDK. The pkg/spdk.spec is considered to be deprecated and scheduled for removal in SPDK 21.07. 1693See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 1694 1695### sock 1696 1697The type of `enable_placement_id` in struct `spdk_sock_impl_opts` is changed from 1698bool to int. We can use RPC to configure different value of `enable_placement_id`. 1699Then we can leverage SO_INCOMING_CPU to get placement_id, which aims to utilize 1700CPU cache locality, enabled by setting enable_placement_id=2. 1701 1702A new socket placement mode called PLACEMENT_MARK has been added. Some NICs allow 1703sockets to be marked using the SO_MARK socket option as a hint for which hardware 1704queue they should be associated with. This mode leverages that by setting the same 1705value for all sockets within a poll group. 1706 1707New parameters `enable_zerocopy_send_server` and `enable_zerocopy_send_client` were added 1708to struct spdk_sock_impl_opts, these parameters enable or disable zero copy send for server 1709and client sockets which are created using `spdk_sock_listen` and `spdk_sock_listen_ext` (server); 1710`spdk_sock_connect` and `spdk_sock_connect_ext` (client) functions. Existing parameter 1711`enable_zerocopy_send` enables or disables zero copy send for both server and client sockets. 1712 1713### thread 1714 1715A new API `spdk_io_channel_get_io_device` was added to get the io_device for the specified 1716I/O channel. 1717 1718Added `spdk_thread_set_interrupt_mode` function in order to set present spdk_thread into 1719interrupt mode or back to poll mode. It is valid only when thread interrupt facility is 1720enabled by `spdk_interrupt_mode_enable`. 1721 1722Added `spdk_poller_register_interrupt` function to mark that the poller is capable of 1723entering interrupt mode. Callback function will be called when the poller must transition 1724into or out of interrupt mode. 1725 1726### virtio 1727 1728Added the `bdev_virtio_blk_set_hotplug` RPC for the virtio blk pci device. 1729 1730## v21.01 1731 1732### bdev 1733 1734An `opts_size` element was added in the `spdk_bdev_opts` structure to solve the 1735ABI compatibility issue between different SPDK version. And also add `opts_size` 1736parameter in spdk_bdev_get_opts function. Two fields `small_buf_pool_size` and 1737`large_buf_pool_size` were added into spdk_bdev_opts, which were used to determine 1738the small and large buffer pool size of the whole bdev module. 1739 1740A new API `spdk_bdev_wait_for_examine` was added to allow for checking state of 1741examine process. Along with corresponding `bdev_wait_for_examine` RPC, which 1742is now always called during `spdk_bdev_subsystem_config_json` making sure 1743every bdev is ready to be used. 1744 1745A new API `spdk_bdev_io_get_aio_status` was added for getting the status of 1746bdev_io as Linux AIO errno. Also `spdk_bdev_io_complete_aio_status` function 1747and `SPDK_BDEV_IO_STATUS_AIO_ERROR` were added for bdev module to complete 1748a bdev_io with Linux AIO errno. 1749 1750A new API `spdk_bdev_get_module_name` was added for getting block device module name. 1751Also `spdk_bdev_get_module_ctx` was added to obtain a bdev module context for 1752the block device opened by the specified descriptor. 1753 1754Added `max_segment_size` and `max_num_segments` to `spdk_bdev` structure for 1755bdev modules to specify splitting requirements. 1756 1757### blob 1758 1759An `opts_size` element was added in the `spdk_bs_opts` structure to solve the 1760ABI compatibility issue between different SPDK version. And also add `opts_size` 1761parameter in `spdk_bs_opts_init` function. 1762 1763An `opts_size` element was added in the `spdk_blob_opts` structure to solve the 1764ABI compatibility issue between different SPDK version. And also add `opts_size` 1765parameter in `spdk_blob_opts_init` function. 1766 1767An `opts_size` element was added in the `spdk_blob_open_opts` structure to solve the 1768ABI compatibility issue between different SPDK version. And also add `opts_size` 1769parameter in `spdk_blob_open_opts_init` function. 1770 1771### build 1772 1773SPDK now generates pkg-config files to simplify the process of determining which 1774libraries must be linked into an SPDK application. 1775 1776### dpdk 1777 1778Updated DPDK submodule to DPDK 20.11. 1779 1780Removed `--with-igb-uio-driver` configure option. Since DPDK 20.11 igb_uio driver 1781was moved to separate dpdk-kmods repository. If required, users need to refer to 1782dpdk-kmods repository to build the driver. 1783 1784### env 1785 1786The `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts` 1787have been deprecated. The new members are named `pci_allowed`, `pci_blocked` and 1788`main_core` respectively. 1789 1790An `opts_size` element was added in the `spdk_app_opts` structure 1791to solve the ABI compatibility issue between different SPDK version. An `opts_size` 1792parameter is added into `spdk_app_opts_init` function. 1793 1794### event 1795 1796The `pci_whitelist` and `pci_blacklist` members of struct `spdk_app_opts` have been 1797deprecated. The new members are named `pci_allowed` and `pci_blocked` respectively. 1798 1799The `--pci-blacklist` command line option has been deprecated, replaced with 1800`--pci-blocked`. 1801 1802The `--pci-whitelist/-W` command line options have been deprecated, replaced with 1803`--pci-allowed/-A`. 1804 1805Added new experimental `dynamic` scheduler that rebalances idle threads, adjusts CPU frequency 1806using dpdk_governor and turns idle reactor cores to interrupt mode. Please see 1807[scheduler documentation](https://www.spdk.io/doc/scheduler.html) for details. 1808 1809## ioat 1810 1811The PCI BDF whitelist option has been removed from the `ioat_scan_accel_engine` RPC. 1812IOAT PCI functions can still be allowed or denied using SPDK application command 1813line options. 1814 1815### isa-l 1816 1817Updated ISA-L submodule to v2.30.0. 1818 1819### json 1820 1821A new API `spdk_jsonrpc_send_bool_response` was added to allow sending response for 1822writing json bool results into one function. 1823Update API `bdev_nvme_set_options` and add a `keep_alive_timeout_ms` parameter. Now you 1824can specify the `keep_alive_timeout` before creating NVMe bdev. 1825 1826### nbd 1827 1828Change the return type of function `spdk_nbd_stop` from void to int. And update the 1829`spdk_nbd_fini` with two parameters to make its behavior from sync to async. 1830 1831### nvme 1832 1833Directives support was added to the NVMe driver. 1834 1835Two async APIs `spdk_nvme_ctrlr_cmd_directive_receive` and `spdk_nvme_ctrlr_cmd_directive_send` 1836are added for Directive Send and Directive Receive command, respectively. 1837 1838Added a new function `spdk_nvme_ctrlr_reset_subsystem` to perform a NVMe 1839subsystem reset. Note: The NVMe-oF target does not support the subsystem reset yet. 1840 1841Add a new function `spdk_nvme_bytes_to_numd` to transfer bytes to number of 1842dwords. 1843 1844Added a new custom transport `SPDK_NVME_TRANSPORT_VFIOUSER` to enable NVMe 1845driver running with NVMe over vfio-user target. 1846 1847Added the vfio-user custom transport implementation in NVMe driver which can connect 1848to NVMe over vfio-user target via vfio-user transport. 1849 1850Added a new function `spdk_nvme_ctrlr_set_remove_cb` to remove or override 1851`remove_cb` and `remove_ctx` specified when the controller was probed. 1852 1853Added support for ZNS zone append command with new API `spdk_nvme_zns_zone_append` and 1854`spdk_nvme_zns_zone_append_with_md`. 1855 1856### nvmf 1857 1858`nvmf_fc_lld_fini` now takes callback and hence updating FC Broadcom LLD driver 1859to the latest is required. 1860 1861FC transport supported primary tagging and VMID. 1862 1863Broadcom FC LLD driver and SPDK NVMe-oF FC transport consolidated one LLD API, 1864`nvmf_fc_init_poller_queues` into another LLD API `nvmf_fc_init_q`. 1865Hence updating Broadcom FC LLD driver to the latest is required. 1866 1867The functions `destroy` and `qpair_fini` in the transport interface now accept a 1868`cb_fn` and `cb_arg` to call upon completion, and their execution can be asynchronous. 1869 1870The SPDK NVMe-oF target now supports async event notification for discovery log changes. 1871This allows the initiator to create persistent connection to discovery controller and 1872be notified of any discovery log changes. 1873 1874An `opts_size` element was added in the `spdk_nvmf_transport_opts` structure 1875to solve the ABI compatibility issue between different SPDK version. And also add 1876`opts_size` parameter in `spdk_nvmf_transport_opts_init` function. 1877 1878Added a new custom vfio-user transport implementation in NVMe-oF which can provide 1879emulated NVMe devices to QEMU and SPDK NVMe driver. 1880 1881Added new API `spdk_nvmf_tgt_listen_ext` that uses `spdk_nvmf_listen_opts` to allow 1882passing generic options to specific transport. This functionality replaces one provided 1883with `nvmf_subsystem_set_options`. Also removed `nvmf_subsystem_set_options` RPC 1884that allowed transport specific options within subsystem. 1885 1886The `trsvcid` in `nvmf_subsystem_add_listener` RPC is now optional. 1887 1888Pausing a subsystem now only pauses all admin queues. This allows for the 1889management of hosts and listeners, plus the addition of namespaces without a 1890full subsystem pause. Additionally, the target now allows for pausing 1891individual namespaces within a subsystem. To remove a namespace from a 1892subsystem, only the specific namespace must be paused. I/O will continue to 1893other namespaces while these operations execute. 1894 1895### rpc 1896 1897An new optional parameter `wait` was added to the RPC `iscsi_create_portal_group`, 1898and an new RPC `iscsi_start_portal_group` was added. They will be used not to start 1899listening on portals for a portal group until all associated target nodes are created 1900at startup, otherwise some iSCSI initiators may fail to re-login when SPDK iSCSI 1901target application restarts. 1902 1903Two optional parameter `--small-buf-pool-size` and `--large-buf-pool-size` were added 1904into `bdev_set_options` function. 1905 1906### vhost 1907 1908Added optional `packed_ring_recovery` parameter to `vhost_create_blk_controller` RPC 1909enabling packed ring live recovery. 1910 1911## v20.10 1912 1913### accel 1914 1915New APIs `spdk_accel_batch_cancel` and `spdk_idxd_batch_cancel` were added to allow 1916cancelling a batch sequence. 1917 1918### bdev 1919 1920A new `spdk_bdev_part_base_construct_ext` function has been added and the 1921`spdk_bdev_part_base_construct` has been deprecated. The 1922`spdk_bdev_part_base_construct_ext` function takes bdev name as an argument instead 1923of bdev structure to avoid a race condition that can happen when the bdev is being 1924removed between a call to get its structure based on a name and actually opening it. 1925 1926Removed `spdk_bdev_config_text` function for bdev modules to report legacy config. 1927 1928### blobstore 1929 1930A new `spdk_bdev_create_bs_dev_ext` function has been added and `spdk_bdev_create_bs_dev_from_desc` 1931function has been deprecated. 1932 1933### dpdk 1934 1935Updated DPDK submodule to DPDK 20.08. 1936 1937### env 1938 1939A new API `spdk_mem_get_fd_and_offset` was added to get file descriptor and offset for 1940a given virtual address. 1941 1942### event 1943 1944Removed `spdk_subsystem_config` callback for submodules as part of legacy config removal. 1945 1946Removed `spdk_app_get_running_config` function that printed configuration in legacy format, 1947and removed `usr1_handler` from `struct spdk_app_opts` callback that was used to call it. 1948 1949Added SPDK thread scheduler framework which is used to rebalance load among CPU cores. 1950The scheduler implementation is pluggable and two schedulers are added first, `static` scheduler 1951and `gscheduler` scheduler. 1952 1953`static` scheduler is used by default and does not reschedule threads and keeps the initial 1954assignments. 1955 1956CPU frequency governor framework and implementation are also added. 1957`dpdk_governor` is the first implementation and is based on rte_power library of DPDK. 1958`gscheduler` scheduler utilizes the CPU frequency governor. 1959 1960Scheduler and governor frameworks are experimental features. 1961 1962### fio 1963 1964Bdev fio_plugin no longer supports legacy configuration files. Options `spdk_conf` and 1965`spdk_json_conf` have to point to valid JSON configuration. 1966 1967### gpt 1968 1969Removed option to disable GPT probing via legacy configuration. Please use generic 1970bdev functionality to replicate the behavior. See `bdev_set_options --disable-auto-examine` 1971and `bdev_examine` RPC. 1972 1973### intel-ipsec-mb 1974 1975Updated intel-ipsec-mb submodule to v0.54 1976 1977### isa-l 1978 1979Updated ISA-L submodule to v2.29.0. 1980 1981### iscsi 1982 1983The iSCSI target supports iSCSI login redirection feature to achieve fault tolerance 1984and load balancing particularly in scale out iSCSI target system, which runs multiple 1985SPDK iSCSI targets. 1986 1987### json 1988 1989A new API `spdk_json_decode_object_relaxed` was added to allow parsing json values 1990even if decoder for a given key is not found. 1991 1992A new API `spdk_json_free_object` was added to free memory allocated by `spdk_json_decode_object`. 1993 1994### log 1995 1996A log flag structure, `spdk_log_register_flag`, `spdk_log_get_first_flag`, 1997`spdk_log_get_next_flag` and macro `SPDK_LOG_REGISTER_COMPONENT`, have been added which 1998enable registering and interacting with log flags. Added `SPDK_INFOLOG`, `SPDK_DEBUGLOG` 1999and `SPDK_LOGDUMP` macros to display logs for specified log flag. 2000 2001The log flags in the event framework `-L` and several SPDK applications has been enabled on 2002release builds of SPDK. On debug builds this option will additionally set 2003log print level to `SPDK_LOG_DEBUG`. 2004 2005A new API `spdk_log_enable_timestamps` and new RPC `log_enable_timestamps` was added to 2006toggle log timestamps. 2007 2008### miscellaneous 2009 2010The contents of the log_rpc library have been moved to the event library. The log_rpc 2011library now no longer exists. 2012 2013The contents of the app_rpc library have been moved to the event library. The app_rpc 2014library now no longer exists. 2015 2016The contents of the bdev_rpc library have been moved to the bdev library. The app_rpc 2017library now no longer exists. 2018 2019The bdevperf application now disables the zcopy API by default. Prior to this change, 2020bdevperf enabled using the zcopy API by default which caused a performance impact of 2021up to 25% on bdevs that don't natively support zcopy because the API emulates zero-copy 2022by allocating a buffer. The bdevperf `-x` param was renamed to `-Z` and the default 2023value changed to false. For bdevs that support zcopy, use the -Z flag to enable 2024using zcopy API. 2025 2026### net 2027 2028Net framework intended for use by user-space TCP stacks was removed. 2029 2030### nvme 2031 2032Zoned Namespace Command Set support has been added to NVMe driver, see `nvme_zns.h` for details. 2033 2034New APIs, `spdk_nvme_detach_async` and `spdk_nvme_detach_poll_async`, have been added to 2035detach multiple controllers in parallel to mitigate lengthy shutdown notification of 2036a few NVMe SSDs. 2037 2038New APIs, `spdk_nvme_ns_get_ana_group_id` and `spdk_nvme_ns_get_ana_state`, have been 2039added to get ANA group ID and ANA state of the given namespace, respectively. 2040 2041If no specific command set is requested (by setting the command_set member in the 2042`spdk_nvme_ctrlr_opts` structure), SPDK will automatically select the most appropriate 2043command set based on what the controller supports. 2044 2045### nvmf 2046 2047A new API `spdk_nvmf_subsystem_add_ns_ext` has been added and the API `spdk_nvmf_subsystem_add_ns` 2048has been deprecated. 2049 2050The NVMe-oF target now supports Asymmetric Namespace Access (ANA) Reporting to provide 2051multipath to NVMe-oF initiator. 2052 2053Add `no_wr_batching` parameter to `spdk_nvmf_transport_opts` struct to disable 2054Work Requests batching in RDMA transport. 2055 2056NVMf Target transports can now parse any additional JSON params in the nvmf_create_transport RPC 2057through the JSON context provided via spdk_nvmf_target_opts->transport_specific. 2058 2059A new RPC `nvmf_subsystem_set_options` was added to allow transport specific options within subsystem. 2060 2061A new API `spdk_nvmf_transport_stop_listen_async` was added to stop accepting new connections 2062at the provided address. `nvmf_subsystem_remove_listener` RPC now uses this function. 2063 2064A new API `spdk_nvmf_subsystem_disconnect_host` was added to disconnect all connections 2065originating from the provided hostnqn. `nvmf_subsystem_remove_host` RPC now uses this function. 2066 2067### ocf 2068 2069Updated OCF submodule to v20.03.1 2070 2071### rpc 2072 2073New optional parameters, `enable_placement_id` and `enable_quickack` were added to the 2074`sock_impl_set_options` RPC. 2075 2076A new RPC `bdev_examine_bdev` was added to allow users to examine a bdev explicitly. 2077It can be used only if bdev_auto_examine is set to false by the RPC `bdev_set_options`. 2078 2079New optional parameters `no_wr_batching` and `control_msg_num` were added to the RPC 2080'nvmf_create_transport'. 2081 2082New RPCs, `iscsi_target_node_set_redirect` and `iscsi_target_node_request_logout`, have 2083been added, and a new optional parameter `private` has been added to the RPC 2084`iscsi_create_portal_group` to support iSCSI login redirection feature. 2085 2086A new optional parameter `ana_reporting` was added to the RPC `nvmf_create_subsystem`, and 2087a new RPC `nvmf_subsystem_listener_set_ana_state` was added for ANA reporting. 2088 2089New RPCs, `nvmf_subsystem_get_listeners` and `nvmf_subsystem_get_qpairs`, were added to 2090retrieve configuration of the NVMe-oF subsystem. 2091 2092### scsi 2093 2094Two new APIs have been added `spdk_scsi_dev_construct_ext` and 2095`spdk_scsi_dev_add_lun_ext` that allow the upper layer(e.g. vhost-scsi) to 2096receive the notification when the scsi bdev has been resized. 2097 2098The `spdk_scsi_dev_construct` and `spdk_scsi_dev_add_lun` eventually may be 2099deprecated and removed. 2100 2101### sock 2102 2103The `enable_placement_id` field was added in the `struct spdk_sock_impl_opts` to 2104make the placement_id feature configurable by users. The default setting is 2105not enabled. 2106 2107The `enable_quick_ack` field was added in the `struct spdk_sock_impl_opts` to enable 2108or disable quick ack for the POSIX sock module. The default setting is not enabled. 2109 2110### thread 2111 2112`fd_group` is applied to support interrupt mode. 2113New APIs were added to support an experimental interrupt mode. This allows modules or 2114libraries to selectively register file descriptors that the spdk_thread can wait on, 2115as an alternative to polling. In v20.10, this functionality is enabled in a very small 2116subset of SPDK libraries and modules. 2117 2118### util 2119 2120A new utility named `fd_group` was add. It is now 2121implemented by epoll on Linux platform. It can be used by 2122spdk_thread and reactor to implement interrupt mode. 2123 2124Added `spdk_bit_pool` utility, see `bit_pool.h` for details. 2125 2126### vhost 2127 2128SPDK has switched to DPDK's rte_vhost library since 19.07 release, removed the internal 2129rte_vhost library which is used for DPDK older than 19.05, removed the experimental vhost 2130nvme target which depends on the internal rte_vhost library. 2131 2132### vpp 2133 2134Removed socket abstraction for VPP. 2135 2136## v20.07: SPDK CSI driver, new accel_fw commands, I/O abort support 2137 2138spdk_json_free_object() was added to free memory allocated by spdk_json_decode_object(). 2139 2140### accel 2141 2142A new API was added `spdk_accel_get_capabilities` that allows applications to 2143query the capabilities of the currently enabled accel engine back-end. 2144 2145A new capability, CRC-32C, was added via `spdk_accel_submit_crc32c`. 2146 2147The software accel engine implementation has added support for CRC-32C. 2148 2149A new capability, compare, was added via `spdk_accel_submit_compare`. 2150 2151The software accel engine implementation has added support for compare. 2152 2153Several APIs were added to `accel_engine.h` to support batched submission 2154of operations. 2155 2156Several APIs were added to `accel_engine.h` to support dualcast operations. 2157 2158### accel_fw 2159 2160The accel_fw was updated to no longer require the app to allocate an 2161accel_task on its behalf. All public APIs now take a callback arg as 2162the parameter that used to be the accel_task. 2163 2164The accel_fw API `spdk_accel_task_size` was removed as it is no longer 2165required. 2166 2167The accel_fw was updated to support compare, dualcast, crc32c. 2168 2169The accel_fw introduced batching support for all commands in all plug-ins. 2170See docs for detailed information. 2171 2172### bdev 2173 2174A new API `spdk_bdev_abort` has been added to submit abort requests to abort all I/Os 2175whose callback context match to the bdev on the given channel. 2176 2177### build 2178 2179The fio plugins now compile to `build/fio` and are named `spdk_bdev` and `spdk_nvme`. 2180Existing fio configuration files will need to be updated. 2181 2182### dpdk 2183 2184Updated DPDK submodule to DPDK 20.05. 2185 2186### env 2187 2188Several new APIs have been added to provide greater flexibility in registering and 2189accessing polled mode PCI drivers. See `env.h` for more details. 2190 2191### idxd 2192 2193The idxd library and plug-in module for the accel_fw were updated to support 2194all accel_fw commands as well as batching. Batching is supported both 2195through the library and the plug-in module. 2196 2197IDXD engine support for CRC-32C has been added. 2198 2199### ioat 2200 2201A new API `spdk_ioat_get_max_descriptors` was added. 2202 2203### nvme 2204 2205An `opts_size`element was added in the `spdk_nvme_ctrlr_opts` structure 2206to solve the ABI compatibility issue between different SPDK version. 2207 2208A new API `spdk_nvme_ctrlr_cmd_abort_ext` has been added to abort previously submitted 2209commands whose callback argument match. 2210 2211Convenience functions, `spdk_nvme_print_command` and `spdk_nvme-print_completion` were added 2212to the public API. 2213 2214A new function, `spdk_nvmf_cuse_update_namespaces`, updates the cuse representation of an NVMe 2215controller. 2216 2217A new function `qpair_iterate_requests` has been added to the nvme transport interface. ALl 2218implementations of the transport interface will have to implement that function. 2219 2220### nvmf 2221 2222The NVMe-oF target no longer supports connecting scheduling configuration and instead 2223always uses what was previously called "transport" scheduling. 2224 2225`spdk_nvmf_tgt_accept` no longer exists. The accept process now occurs automatically after 2226the creation of an nvmf target and queue pairs are assigned to poll groups by the underlying 2227transport. Further, `spdk_nvmf_transport_ops` has changed such that the accept function 2228pointer no longer takes a function pointer as an argument. Instead, transports should call 2229`spdk_nvmf_tgt_new_qpair` whenever they previously would have called that callback. 2230 2231The NVMe-oF target now supports aborting any submitted NVM or Admin command. Previously, 2232the NVMe-oF target could abort only Asynchronous Event Request commands. 2233 2234### rdma 2235 2236A new `rdma` library has been added. It is an abstraction layer over different RDMA providers. 2237Two providers are available - verbs (used by default when RDMA is enabled or enabled explicitly 2238using --with-rdma=verbs) and mlx5 Direct Verbs aka DV (enabled by --with-rdma=mlx5_dv). 2239Using mlx5_dv requires libmlx5 installed on the system. 2240 2241### rpc 2242 2243Parameter `-p` or `--max-qpairs-per-ctrlr` of `nvmf_create_transport` RPC command accepted by the 2244rpc.py script is deprecated, new parameter `-m` or `--max-io-qpairs-per-ctrlr` was added. 2245 2246Added `sock_impl_get_options` and `sock_impl_set_options` RPC methods. 2247 2248Command line parameters `-r` and `--rpc-socket` will longer accept TCP ports. RPC server 2249must now be started on a Unix domain socket. Exposing RPC on the network, as well as providing 2250proper authentication (if needed) is now a responsibility of the user. 2251 2252The `bdev_set_options` RPC has a new option, `bdev_auto_examine` to control the auto examine function 2253of bdev modules. 2254 2255New RPCs `sock_impl_get_options` and `sock_impl_set_options` been added to expose new socket features. 2256See `sock` section for more details. 2257 2258### sock 2259 2260Added `spdk_sock_impl_get_opts` and `spdk_sock_impl_set_opts` functions to set/get socket layer configuration 2261options. Options can be set independently for each implementation. 2262 2263Added `recv_buf_size` and 'send_buf_size' socket layer options. They are used only in posix implementation. 2264 2265Added `uring` based socket implementation, the code is located in module/sock/uring. This feature is only 2266available in Linux which requires kernel version is greater than 5.4.3. Currently, our CI pool added the uring 2267based socket tests for iSCSI target and also the tests for SPDK NVMe-oF tcp transport. 2268 2269Added `enable_recv_pipe` socket layer option to allow disabling of double buffering on receive. 2270New option is used only in posix implementation. 2271 2272Added `enable_zerocopy_send` socket layer option to allow disabling of zero copy flow on send. 2273New option is used only in posix implementation. 2274 2275### util 2276 2277Some previously exposed CRC32 functions have been removed from the public API - 2278`spdk_crc32_update`, `spdk_crc32_table_init`, and the `spdk_crc32_table` struct. 2279 2280### vhost 2281 2282The function `spdk_vhost_blk_get_dev` has been removed. 2283 2284## v20.04: SPDK Top, IDXD, NVMe qpair groups 2285 2286IDXD engine support for compare has been added. 2287 2288### configuration 2289 2290Legacy INI style configuration for SPDK applications has been deprecated and will be 2291removed in future release. 2292Please switch to JSON-RPC configuration files and/or RPC driven run-time configuration. 2293 2294### copy 2295 2296The copy engine library, modules and public APIs have been renamed. Use of the word `copy` 2297has been replaced with the word `accel` short for accelerator in preparation for adding new 2298capabilities in the future. Additionally, APIs for what was previously called the `memcpy` 2299engine have been renamed to identify the engine as a software accelerator. 2300 2301### crypto 2302 2303Support for AES_XTS was added for the QAT polled mode driver (pmd). The create RPC 2304`bdev_crypto_create` has 2 new optional parameters: cipher and key2. Cipher can be either 2305AES_CBC (default) or AES_XTS. AES_XTS is only valid when using the QAT polled mode driver. 2306The key2 parameter is the second key required for AES_XTS. 2307 2308### event 2309 2310Reactors now accumulate CPU stats and they are retrieved by the RPC `framework_get_reactors`. 2311 2312### idxd 2313 2314IDXD support was added in the form of a low level library that can directly 2315utilize the IDXD offload engine as well as an accel plug-in module allowing for use 2316with the generic accel framework API. IDXD is the first in a family of offload 2317engines that share the same interface, specifically DSA is added here. More info 2318can be found here: https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator 2319 2320Much of the implementation models IOAT, however the low level interfaces are very 2321different. The RPC to enable IDXD requires a configuration number as well. The 2322code includes two pre-defined configurations of IDXD groups/work queues/engines. A future 2323version will provide an interface to allow for setting of individual configuration 2324parameters. 2325 2326IDXD is not yet available so this feature should be considered experimental. It will 2327be built up with additional documentation as an ongoing activity. 2328 2329### iscsi 2330 2331The iSCSI target now creates a lightweight thread per poll group instead of assuming a pool 2332of lightweight threads already exist at start up time. A poll group is a collection of 2333unrelated iSCSI connections. Each poll group is only accessed from the associated 2334lightweight thread. 2335 2336### ftl 2337 2338Several changes have been made to the `spdk_ftl_conf`, `spdk_ftl_dev_init_ops`, and 2339`spdk_ftl_attrs` structs. Please see `include/spdk/ftl.h` for more details. 2340 2341### miscellaneous 2342 2343The `--json-ignore-init-errors` command line parameter has been added to ignore 2344initialization errors on JSON config load. 2345 2346The public header file io_channel.h has been removed. Please use thread.h which has the 2347exact same API. 2348 2349### nvme 2350 2351Exported internal nvme_ctrlr_cmd_security_receive/send() APIs as public APIs with "the spdk_" 2352prefix. 2353 2354Added `priority` field in `spdk_nvme_transport_id`, this field is used to specify the priority 2355of the NVMe-oF connection, and currently it is used for NVMe-oF tcp connection. 2356 2357A new poll group API has been added to allow for pooling of nvme qpairs across a single 2358entity which can be polled for completions. This new API consists of the `spdk_nvme_poll_group` 2359family of functions. As a result of this new API, all NVMe transports are expected to implement 2360several poll group related functions. 2361 2362A new flag, `create_only`, has been added to the `spdk_nvme_io_qpair_opts` structure. This flag 2363allows a user to call `spdk_nvme_ctrlr_get_default_io_qpair` without also connecting the qpair 2364within the context of that call. 2365 2366As a result of the `create_only` flag, two new API functions, `spdk_nvme_ctrlr_connect_io_qpair` 2367and `spdk_nvme_ctrlr_disconnect_io_qpair`, have been added to facilitate connecting newly created 2368qpairs (for example, after they have been added to a poll group) and disconnecting qpairs without 2369destroying them (for example to disconnect a qpair before migrating it to a new poll group and 2370reconnecting it). 2371 2372The functions `spdk_nvme_ctrlr_alloc_cmb_io_buffer` and `spdk_nvme_ctrlr_free_cmb_io_buffer` 2373have been changed to `spdk_nvme_ctrlr_map_cmb` and `spdk_nvme_ctrlr_unmap_cmb` respectively. 2374 2375An additional function, `spdk_nvme_ctrlr_reserve_cmb`, has been added to facilitate reserving 2376the entire size of the controller memory buffer for data transfer. 2377 2378### nvme_cuse 2379 2380`spdk_nvme_cuse_get_ctrlr_name` now takes two additional parameters, `char *name` which 2381stores the pointer to the controller name, and `size_t *size` which stores the length of 2382the name. The return type has also been changed from char * to int. 2383 2384`spdk_nvme_cuse_get_ns_name` now takes two additional parameters, `char *name` which 2385stores the pointer to the namespace name, and `size_t *size` which stores the length of 2386the name. The return type has also been changed from char * to int. 2387 2388### nvme_opal 2389 2390Several public OPAL structure definitions have been changed since the last release. 2391 2392### nvmf 2393 2394`spdk_nvmf_poll_group_destroy()` is now asynchronous and accepts a completion callback. 2395 2396The NVMe-oF target now creates a lightweight thread per poll group instead of assuming a pool 2397of lightweight threads already exist at start up time. A poll group is a collection of 2398unrelated NVMe-oF connections. Each poll group is only accessed from the associated 2399lightweight thread. 2400 2401A new struct, `spdk_nvmf_subsystem_listener`, has been added to encapsulate the subsystem specific 2402nature of a listener object. 2403 2404`spdk_nvmf_tgt_listen` no longer accepts a callback function or argument. It also returns an 2405int to indicate the status of the listen call. 2406 2407The execution of `spdk_nvme_poll_group_destroy` is now asynchronous and the function accepts 2408a cb_fn and cb_arg to call upon completion. 2409 2410The execution of `spdk_nvmf_subsystem_add_listener` is now asynchronous and the function accepts 2411a cb_fn and cb_arg to call upon completion. 2412 2413The `nvmf_transport.h` header has been made public to allow custom NVMe-oF transports to integrate 2414with NVMe-oF libraries without using internal APIs. 2415 2416### ocf 2417 2418Updated the OCF submodule to OCF v20.03 2419 2420New version of OCF comes with API changes and bug fixes 2421 2422### rpc 2423 2424A new RPC `thread_set_cpumask` has been added to set the cpumask of the thread 2425to the specified value. 2426 2427A new RPC `thread_get_pollers` has been added to retrieve pollers from SPDK threads. 2428 2429A new RPC `thread_get_io_channels` has been added to retrieve I/O channels from SPDK threads. 2430 2431A new RPC `bdev_rbd_resize` has been added to resize Ceph RBD bdevs. 2432 2433### sock 2434 2435The `spdk_sock_set_priority` function has been removed since the feature to set the sock priority 2436will be contained in two new functions, i.e., `spdk_sock_listen_ext` and `spdk_sock_connect_ext`. 2437Users may now specify the priority of the socket in the opts that they want to use. 2438 2439### spdk_top 2440 2441A new application, `spdk_top`, has been added which allows users to monitor resource consumption 2442by a running SPDK application. More information on this application can be found in 2443`app/spdk_top/README`. 2444 2445### thread 2446 2447A new function `spdk_thread_lib_init_ext` has been added, and the function 2448`spdk_thread_lib_init` has been deprecated. The user of `spdk_thread_lib_init_ext` is 2449expected to implement both functions `spdk_thread_op_fn` and `spdk_thread_op_supported_fn`. 2450`spdk_thread_op_supported_fn` is called to check whether the SPDK thread operation 2451is supported. `spdk_thread_op_fn` is called to execute the SPDK thread operation. 2452Current SPDK operation types are `SPDK_THREAD_OP_NEW` and `SPDK_THREAD_OP_RESCHED`. 2453The operation `SPDK_THREAD_OP_NEW` is called each time a new thread is created. 2454The operation `SPDK_THREAD_OP_RESCHED` is called when SPDK thread needs to be rescheduled. 2455 2456A unique ID has been added for each created SPDK thread, it is retrieved by a new function 2457`spdk_thread_get_id`, and the SPDK thread which has the specific ID is got by 2458a new function `spdk_thread_get_by_id`. 2459 2460A new function `spdk_thread_cpumask` has been added to set the current thread's cpumask 2461to the specified value. The function requires the operation `SPDK_THREAD_OP_RESCHED` 2462is supported. 2463 2464A new function `spdk_poller_register_named` has been added to set arbitrary name to the 2465created poller. If NULL, the name is set to the pointer of the poller function. 2466 2467The function `spdk_thread_poll` now measures run time per call correctly on multiple SPDK 2468threads configuration, and a new function `spdk_thread_get_last_tsc` has been added to use together. 2469 2470Voluntary termination of SPDK thread has been supported by refining the functions `spdk_thread_exit` 2471and `spdk_thread_poll`. 2472 2473### util 2474 2475New functions `spdk_sn32_lt` and `spdk_sn32_gt` have been added. They compare two sequence 2476numbers based on serial number arithmetic. 2477 2478### vhost 2479 2480Poll groups per session have been replaced by SPDK threads per vhost controller. 2481 2482### vmd 2483 2484A new function, `spdk_vmd_fini`, has been added. It releases all resources acquired by the VMD 2485library through the `spdk_vmd_init` call. 2486 2487## v20.01: Optimized thin provisioning, FTL bdev, VMD hot plug, FUSED support 2488 2489### bdev 2490 2491A new function, `spdk_bdev_set_timeout`, has been added to set per descriptor I/O timeouts. 2492 2493A new class of functions `spdk_bdev_compare*`, have been added to allow native bdev support 2494of block comparisons and compare-and-write. 2495 2496A new class of bdev events, `SPDK_BDEV_EVENT_MEDIA_MANAGEMENT`, has been added to allow bdevs 2497which expose raw media to alert all I/O channels of pending media management events. 2498 2499A new API was added `spdk_bdev_io_get_aux_buf` allowing the caller to request 2500an auxiliary buffer for its own private use. The API is used in the same manner that 2501`spdk_bdev_io_get_buf` is used and the length of the buffer is always the same as the 2502bdev_io primary buffer. 'spdk_bdev_io_put_aux_buf' frees the allocated auxiliary 2503buffer. 2504 2505### blobfs 2506 2507Added boolean return value for function spdk_fs_set_cache_size to indicate its operation result. 2508 2509Added `blobfs_set_cache_size` RPC method to set cache size for blobstore filesystem. 2510 2511### blobstore 2512 2513Added new `use_extent_table` option to `spdk_blob_opts` for creating blobs with Extent Table descriptor. 2514Using this metadata format, dramatically decreases number of writes required to persist each cluster allocation 2515for thin provisioned blobs. Extent Table descriptor is enabled by default. 2516See the [Blobstore Programmer's Guide](https://spdk.io/doc/blob.html#blob_pg_cluster_layout) for more details. 2517 2518### dpdk 2519 2520Updated DPDK submodule to DPDK 19.11. 2521 2522### env_dpdk 2523 2524`spdk_env_dpdk_post_init` now takes a boolean, `legacy_mem`, as an argument. 2525 2526A new function, `spdk_env_dpdk_dump_mem_stats`, prints information about the memory consumed by DPDK to a file specified by 2527the user. A new utility, `scripts/dpdk_mem_info.py`, wraps this function and prints the output in an easy to read way. 2528 2529### event 2530 2531The functions `spdk_reactor_enable_framework_monitor_context_switch()` and 2532`spdk_reactor_framework_monitor_context_switch_enabled()` have been changed to 2533`spdk_framework_enable_context_switch_monitor()` and 2534`spdk_framework_context_switch_monitor_enabled()`, respectively. 2535 2536### ftl 2537 2538All NVMe dependencies were removed from ftl library. 2539From now ftl library is using bdev_zone API. 2540bdev_ftl becomes virtual bdev. 2541 2542`ctrlr` and `trid` fields in `spdk_ftl_dev_init_opts` structure were replaced by 2543`base_bdev_desc`. 2544 2545`bdev_ftl_create` RPC method `trtype` and `traddr` parameters were replaced by `base_bdev` 2546parameter. 2547 2548`spdk_ftl_module_init` and `spdk_ftl_module_fini` functions were removed. 2549 2550`spdk_ftl_punit_range` and `ftl_module_init_opts` structures were removed. 2551 2552### isa-l 2553 2554Updated ISA-L submodule to commit f3993f5c0b6911 which includes implementation and 2555optimization for aarch64. 2556 2557Enabled ISA-L on aarch64 by default in addition to x86. 2558 2559### nvme 2560 2561`delayed_pcie_doorbell` parameter in `spdk_nvme_io_qpair_opts` was renamed to `delay_cmd_submit` 2562to allow reuse in other transports. 2563 2564Added RDMA WR batching to NVMf RDMA initiator. Send and receive WRs are chained together 2565and posted with a single call to ibv_post_send(receive) in the next call to qpair completion 2566processing function. Batching is controlled by 'delay_cmd_submit' qpair option. 2567 2568The NVMe-oF initiator now supports plugging out of tree NVMe-oF transports. In order 2569to facilitate this feature, several small API changes have been made: 2570 2571The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 2572A new function, `spdk_nvme_transport_available_by_name`, has been added. 2573A function table, `spdk_nvme_transport_ops`, and macro, `SPDK_NVME_TRANSPORT_REGISTER`, have been added which 2574enable registering out of tree transports. 2575 2576A new function, `spdk_nvme_ns_supports_compare`, allows a user to check whether a given namespace supports the compare 2577operation. 2578 2579A new family of functions, `spdk_nvme_ns_compare*`, give the user access to submitting compare commands to NVMe namespaces. 2580 2581A new function, `spdk_nvme_ctrlr_cmd_get_log_page_ext`, gives users more granular control over the command dwords sent in 2582log page requests. 2583 2584### nvmf 2585 2586Add SockPriority option in [Transport] section, this can be used for NVMe-oF target 2587on TCP transport to set sock priority for the incoming TCP connections. 2588 2589The NVMe-oF target now supports plugging out of tree NVMe-oF transports. In order 2590to facilitate this feature, several small API changes have been made: 2591 2592The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 2593`spdk_nvmf_tgt_get_transport`, `spdk_nvmf_transport_opts_init`, and `spdk_nvmf_transport_create` all have been 2594modified to take a string. 2595A function table, `spdk_nvmf_transport_ops`, and macro, `SPDK_NVMF_TRANSPORT_REGISTER`, have been added which 2596enable registering out of tree transports. 2597 2598Add `spdk_nvmf_tgt_stop_listen()` that can be used to stop listening for 2599incoming connections for specified target and trid. Listener is not stopped 2600implicitly upon destruction of a subsystem any more. 2601 2602A custom NVMe admin command handler has been added which allows the user to use the real drive 2603attributes from one of the target NVMe drives when reporting drive attributes to the initiator. 2604This handler can be enabled via the `nvmf_set_config` RPC. 2605Note: In a future version of SPDK, this handler will be enabled by default. 2606 2607The SPDK target and initiator both now include compare-and-write functionality with one caveat. If using the RDMA transport, 2608the target expects the initiator to send both the compare command and write command either with, or without inline data. The 2609SPDK initiator currently respects this requirement, but this note is included as a flag for other initiators attempting 2610compatibility with this version of SPDK. 2611 2612The `spdk_nvme_ctrlr_opts` struct has been extended with new field `transport_ack_timeout` which allows 2613to configure transport ACK timeout. Applicable for RDMA transport only. 2614 2615### rpc 2616 2617A new RPC, `bdev_zone_block_create`, enables creating an emulated zoned bdev on top of a standard block device. 2618 2619A new RPC, `bdev_ocssd_create`, enables creating an emulated zoned bdev on top of an Open Channel SSD. 2620 2621A new RPC, `blobfs_set_cache_size`, enables managing blobfs cache size. 2622 2623A new RPC, `env_dpdk_get_mem_stats`, has been added to facilitate reading DPDK related memory 2624consumption stats. Please see the env_dpdk section above for more details. 2625 2626A new RPC, `framework_get_reactors`, has been added to retrieve a list of all reactors. 2627 2628`bdev_ftl_create` now takes a `base_bdev` argument in lieu of `trtype`, `traddr`, and `punits`. 2629 2630`bdev_nvme_set_options` now allows users to disable I/O submission batching with the `-d` flag 2631 2632`bdev_nvme_cuse_register` now accepts a `name` parameter. 2633 2634`bdev_uring_create` now takes arguments for `bdev_name` and `block_size` 2635 2636`nvmf_set_config` now takes an argument to enable passthru of identify commands to base NVMe devices. 2637Please see the nvmf section above for more details. 2638 2639### scsi 2640 2641`spdk_scsi_lun_get_dif_ctx` now takes an additional argument of type `spdk_scsi_task`. 2642 2643### sock 2644 2645Added spdk_sock_writev_async for performing asynchronous writes to sockets. This call will 2646never return EAGAIN, instead queueing internally until the data has all been sent. This can 2647simplify many code flows that create pollers to continue attempting to flush writes 2648on sockets. 2649 2650Added `impl_name` parameter in spdk_sock_listen and spdk_sock_connect functions. Users may now 2651specify the sock layer implementation they'd prefer to use. Valid implementations are currently 2652"vpp" and "posix" and NULL, where NULL results in the previous behavior of the functions. 2653 2654### thread 2655 2656`spdk_thread_send_msg` now returns int indicating if the message was successfully 2657sent. 2658 2659A new function `spdk_thread_send_critical_msg`, has been added to support sending a single message from 2660a context that may be interrupted, e.g. a signal handler. 2661 2662Two new functions, `spdk_poller_pause`, and `spdk_poller_resume`, have been added to give greater control 2663of pollers to the application owner. 2664 2665### util 2666 2667`spdk_pipe`, a new utility for buffering data from sockets or files for parsing 2668has been added. The public API is available at `include/spdk/pipe.h`. 2669 2670### bdev 2671 2672Added spdk_bdev_io_get_nvme_fused_status function for translating bdev_io status to NVMe status 2673code for fused compare-and-write operation. 2674 2675Added spdk_bdev_get_acwu function for getting block device atomic compare and write unit size. 2676 2677## v19.10: Zoned bdev API, Opal bdev, NVMe character devices 2678 2679### rpc 2680 2681Many of SPDK's RPCs were renamed to be more consistent and intuitive in this release. 2682The old names will continue to function, but will display a deprecation warning. 2683 2684Added optional parameters '--arbitration-burst' and '--low/medium/high-priority-weight' to 2685'bdev_nvme_set_options' RPC method. 2686 2687Added optional parameter '--md-size' to 'construct_null_bdev' RPC method. 2688 2689Added optional parameters '--dif-type' and '--dif-is-head-of-md' to 'construct_null_bdev' 2690RPC method. 2691 2692Added `blobfs_detect` RPC method to detect whether a blobfs exists on given bdev. 2693 2694Added `blobfs_create` RPC method to build blobfs on given bdev. 2695 2696Added `blobfs_mount` RPC method to mount blobfs on given bdev to a host path by FUSE. 2697Then on the host path, user can directly do some file operations which will be mapped 2698to blobfs. 2699 2700### bdev 2701 2702Added new parameter `cdw0` to `spdk_bdev_io_complete_nvme_status()` and 2703`spdk_bdev_io_get_nvme_status()` that allows setting/getting 2704the NVMe completion queue DW0 entry. This allows vendor specific IO commands 2705to return command specific completion info back to the initiator. 2706 2707Added `spdk_bdev_get_write_unit_size()` function for retrieving required number 2708of logical blocks for write operation. 2709 2710New zone-related fields were added to the result of the `get_bdevs` RPC call: 2711 2712- `zoned`: indicates whether the device is zoned or a regular 2713 block device 2714- `zone_size`: number of blocks in a single zone 2715- `max_open_zones`: maximum number of open zones 2716- `optimal_open_zones`: optimal number of open zones 2717 2718The `zoned` field is a boolean and is always present, while the rest is only available for zoned 2719bdevs. 2720 2721A new `spdk_bdev_open_ext` function has been added and `spdk_bdev_open` function has been deprecated. 2722The new open function introduces requirement to provide callback function that will be called by 2723asynchronous event such as bdev removal. `spdk_bdev_open_ext` function takes bdev name as 2724an argument instead of bdev structure to avoid a race condition that can happen when the bdev 2725is being removed between a call to get its structure based on a name and actually opening it. 2726 2727New 'resize' event has been added to notify about change of block count property of block device. 2728Event is delivered only if block device was opened with `spdk_bdev_open_ext` function. 2729 2730### bdev zone 2731 2732Added new public header for zoned bdev. Zoned bdev is an extension 2733of the bdev interface. 2734 2735`spdk_bdev_get_zone_size()`, `spdk_bdev_get_max_open_zones()`, `spdk_bdev_get_optimal_open_zones()` 2736APIs were added for retrieving zoned device information. 2737`spdk_bdev_get_zone_info()` API was added for retrieving information about zones in zoned 2738device. 2739Added `spdk_bdev_zone_management()` API for changing zone state. 2740`spdk_bdev_zone_append()` and `spdk_bdev_zone_append_with_md()` APIs were added for 2741appending data to a zone. 2742Added `spdk_bdev_io_get_append location()` function for retrieving append location for I/O. 2743Added `spdk_bdev_is_zoned()` function for checking if bdev supports zoned namespace semantics. 2744 2745### bdev opal 2746 2747EXPERIMENTAL: A new opal bdev has been added to support management of 2748NVMe self-encrypting drives through the Opal specification. Users can 2749create opal bdevs from an NVMe namespace bdev, if the controller 2750containing that namespace supports Opal. Currently this is only 2751supported for namespace ID=1. The following RPCs have been added to 2752support Opal: `bdev_nvme_opal_init`, `bdev_nvme_opal_revert`, 2753`bdev_opal_create`, `bdev_opal_delete`, `bdev_opal_get_info`, 2754`bdev_opal_new_user`, `bdev_opal_set_lock_state`. 2755It does not yet support recreating the opal bdevs after application restart. 2756This bdev module should be considered very experimental, and the RPCs may 2757change significantly in future releases. 2758 2759### delay bdev 2760 2761The `bdev_delay_update_latency` has been added to allow users to update 2762a latency value for a given delay bdev. 2763 2764### compress bdev 2765 2766A new RPC `bdev_compress_get_orphans` has been added to list compress bdevs 2767that were not loaded due to a missing pm metadata file. In this state they 2768can only be deleted. 2769 2770### null bdev 2771 2772Metadata support has been added to Null bdev module. 2773 2774Protection information support has been added to Null bdev module. 2775 2776### nvme 2777 2778Added `no_shn_notification` to NVMe controller initialization options, users can enable 2779it for NVMe controllers. When the option is enabled, the controller will not do the 2780shutdown process and just disable the controller, users can start their application 2781later again to initialize the controller to the ready state. 2782 2783A controller flag `SPDK_NVME_CTRLR_WRR_SUPPORTED` was added to indicate the controller 2784can support weighted round robin arbitration feature with submission queue. 2785 2786Added `arbitration_burst` option for arbitration feature, and added three 2787`low/medium/high_priority_weight` options for weighted round robin arbitration. 2788 2789Added `spdk_nvme_ns_cmd_write_uncorrectable`. 2790 2791Added new error handling and reporting functionality. This includes several 2792new API functions to facilitate applications recovering when a qpair or 2793controller fails. 2794 2795`spdk_nvme_ctrlr_reconnect_io_qpair` - Reconnects a failed I/O qpair. 2796`spdk_nvme_ctrlr_set_trid` - Sets the trid of an existing controller. Can be used to 2797change the trid for failover cases. 2798`spdk_nvme_ctrlr_is_failed` - Returns the failed state of a controller. 2799`spdk_nvme_ctrlr_fail` - Forces a controller into a failed state. 2800 2801Modified the return behavior of several API functions to better indicate to 2802applications when a qpair is failed. This list of functions includes: 2803 2804`spdk_nvme_qpair_process_completions` 2805`spdk_nvme_ns_cmd_*` 2806`spdk_nvme_ctrlr_process_admin_completions` 2807`spdk_nvme_ctrlr_cmd_*` 2808 2809These functions now return -ENXIO when the qpair or controller on which they 2810operate is failed. 2811 2812EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux 2813kernel for controller as well as for namespace and process ioctl requests as usual 2814from linux environment. 2815 2816### nvmf 2817 2818The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts` 2819as its only parameter. This new structure contains the max_subsystems parameter previously 2820passed into that function. 2821 2822A new public API function `spdk_nvmf_get_tgt` has been added which allows users to 2823retrieve a pointer to an `spdk_nvmf_tgt` object by supplying its name. In the special 2824case where an RPC or application only creates a single target, this function can accept 2825a null name parameter and will return the only available target. 2826 2827The majority of the NVMe-oF RPCs now accept an optional tgt_name parameter. This will 2828allow those RPCs to work with applications that create more than one target. 2829 2830Three new NVMe-oF RPCs have been added `nvmf_create_target`, `nvmf_delete_target`, and 2831`nvmf_get_targets`. These new RPCs provide a basic interface for managing multiple target 2832objects. In SPDK the target object defines a unique discovery service. As of this release, 2833these RPCs are not intended to be used with the in-tree SPDK target applications, spdk_tgt and 2834nvmf_tgt, which use a single, global target structure. As such, they are not included in scripts/rpc.py 2835 2836Three new header functions have also been added to help deal with multiple targets. 2837`spdk_nvmf_tgt_get_name` takes a target pointer as an argument and returns its human readable name. 2838`spdk_nvmf_get_first_target` takes no arguments and returns the first target in the global list. 2839`spdk_nvmf_get_next_tgt` takes a target pointer as an argument and returns the next one in the global list. 2840 2841The `spdk_nvmf_tgt_accept` takes additional argument allowing to pass arbitrary context 2842information to the `new_qpair` callback. This will simplify the code when having multiple 2843nvmf targets or when retrieving the context information from globals is not suitable. 2844 2845### blobstore 2846 2847A new `spdk_bdev_create_bs_dev_from_desc` function has been added and `spdk_bdev_create_bs_dev` 2848function has been deprecated. 2849The new create function can cowork with `spdk_bdev_open_ext` function, which provides callback 2850function that will be called by asynchronous event such as bdev removal. 2851 2852### blobfs_bdev 2853 2854A new blobfs module `bdev` has been added to simplify the operations of blobfs on bdev. 2855 2856Function `spdk_blobfs_bdev_detect` is added to detect whether blobfs exists on the given block device. 2857 2858Function `spdk_blobfs_bdev_create` is added to create a blobfs on the given block device. 2859 2860Function `spdk_blobfs_bdev_mount` is added to mount a blobfs on the given block device to 2861a host path by FUSE. Then, a new thread is created dedicatedly for one mountpoint to handle 2862FUSE request by blobfs API. 2863 2864### build 2865 2866Option to build FUSE components into blobfs_bdev module for mounting a blobfs filesystem. 2867It requires the installation of libfuse3. By default, it is disabled. And it will be 2868enabled if run `./configure` with `--with-fuse` option. 2869 2870### iSCSI 2871 2872Portals may no longer be associated with a cpumask. The scheduling of 2873connections is moving to a more dynamic model. 2874 2875An new RPC `iscsi_portal_group_set_auth` has been added to set CHAP authentication 2876for discovery sessions specific for the existing iSCSI portal group. This RPC overwrites 2877the setting by the global parameters for the iSCSI portal group. 2878 2879### socket 2880 2881Added `spdk_sock_is_connected` to check whether the socket is currently connected. 2882`spdk_sock_group_poll` now returns number of events on success. 2883 2884### env 2885 2886Added `spdk_pci_device_unclaim()` function to cleanup pci claim file. 2887 2888### event 2889 2890`framework_start_init` RPC no longer stops the application on error during 2891initialization. 2892 2893### DPDK 2894 2895Updated DPDK submodule to DPDK 19.08. 2896 2897### ocf 2898 2899Updated OCF submodule to OCF v19.06 2900 2901Along with update, new cache mode 'write only' was added. 2902 2903New cache modes added to use via RPC, wi - write invalidate and wa - write around. 2904 2905New version of OCF provides fully asynchronous management API. 2906 2907## v19.07: NVMe-oF FC Transport, VMD, NVMe-oF Persistent reservations, Bdev I/O with separate metadata 2908 2909### ftl 2910 2911EXPERIMENTAL: Added ability to mirror writes to persistent write buffer cache 2912to allow for recovery from dirty shutdown event. 2913 2914Added handling of Asynchronous Nand Management Events (ANM). 2915 2916### vmd 2917 2918EXPERIMENTAL: Added Intel Volume Management Device (VMD) driver. VMD is an integrated 2919controller inside the CPU PCIe root complex. It enables virtual HBAs for the connected 2920NVMe SSDs. `spdk_vmd_init()` enumerates NVMe SSDs behind VMD device and hook them into 2921SPDK PCI subsystem. `spdk_nvme_probe()` or `spdk_nvme_connect()` can be used to connect 2922NVMe driver to the device located at the given transport ID. 2923 2924To obtain transport ID of NVMe SSD behind VMD `spdk_lspci` can be used. 2925 2926Current implementation does not support hotplug. 2927 2928### blobfs 2929 2930Blobfs file asynchronous operations were added to public APIs. 2931 2932### util 2933 2934A new file API `spdk_posix_file_load` was added to load file content into a data buffer. 2935 2936New APIs `spdk_dif_ctx_set_data_offset`, `spdk_dif_verify_stream`, 2937`spdk_dif_get_range_with_md`, `spdk_dif_get_length_with_md` have been added, 2938and existing APIs `spdk_dif_ctx_init`, `spdk_dif_set_md_interleave_iovs`, and 2939`spdk_dif_generate_stream` have been refined to insert or strip DIF into iovs 2940windows of any alignment. 2941 2942New APIs `spdk_dif_ctx_set_remapped_init_ref_tag`, `spdk_dif_remap_ref_tag`, 2943and `spdk_dix_remap_ref_tag` have been added to remap DIF reference tag. 2944 2945New APIs `spdk_dif_update_crc32c` and `spdk_dif_update_crc32c_stream` have been 2946added to compute CRC-32C checksum for extended LBA payload. 2947 2948Bdevperf and bdevio applications now support starting tests with application specific 2949RPCs. Please see helper Python scripts in their respective directories. 2950This is a move towards simpler RPC-only configuration for all main 2951and auxiliary applications. 2952 2953Legacy INI style configuration for SPDK applications will become deprecated in SPDK 19.10, 2954and removed in SPDK 20.01. Please consider moving to JSON-RPC configuration files and/or 2955RPC driven run-time configuration. 2956 2957### nvmf 2958 2959EXPERIMENTAL: A Fibre Channel transport that supports Broadcom HBAs has been 2960added. This depends on the FC HBA driver at 2961https://github.com/ecdufcdrvr/bcmufctdrvr. See [the documentation](https://spdk.io/doc/nvmf.html#nvmf_fc_transport) 2962for more information. 2963 2964Persistent reservation emulation has been added to the NVMe-oF target. Persistent reservation 2965state is stored in a JSON file on the local filesystem between target restart. To support this, 2966an optional parameter to the RPC method `nvmf_subsystem_add_ns` called `--ptpl-file` was added. 2967This allows the user to specify which file to store the persistent reservation state in. Note 2968that this is done per namespace. 2969 2970The c2h success optimization under which a command capsule response is not sent 2971for reads is turned on by default. A config knob was added to allow disabling 2972the optimization. This will mostly be used for integration testing with 5.0.x kernels 2973while some compatibility fixes make their way down the pipeline for 5.1.x kernels. 2974 2975The sock priority setting of the TCP connection owned by the tcp transport is added. It is 2976used to optimize the TCP connection performance under designated traffic classes. And the 2977priority is used to differentiate the sock priority between SPDK NVMe-oF TCP target application 2978and other TCP based applications. 2979 2980Shared receive queue can now be disabled even for NICs that support it using the 2981`nvmf_create_transport` RPC method parameter `no_srq`. The actual use of a shared 2982receive queue is predicated on hardware support when this flag is not used. 2983 2984spdk_nvmf_get_optimal_poll_group was added, which is used to return the optimal 2985poll group for the qpair. And `ConnectionScheduler` configuration is added into the 2986[Nvmf] section in etc/spdk/nvmf.conf.in to demonstrate how to configure the connection 2987scheduling strategy among different spdk threads. 2988 2989Added infrastructure to retrieve global and per poll group NVMf statistics. 2990 2991DIF strip and insert is now supported for TCP transport. When it is enabled, DIF 2992setting is not exposed to the NVMe-oF initiator, and DIF is attached into data 2993for write I/O and stripped from data for read I/O. 2994 2995Added a field `dif_insert_or_strip` to struct spdk_nvmf_transport_opts, and 2996updated the related rpc function nvmf_create_transport to make this 2997configurable parameter available to users. The `dif_insert_or_strip` is relevant 2998for TCP transport for now and used to configure the DIF strip and insert. 2999 3000Added infrastructure to retrieve NVMf transport statistics. 3001 3002### notify 3003 3004The function `spdk_notify_get_types()` and `spdk_notify_get_events()` were 3005renamed to `spdk_notify_foreach_type()` and `spdk_notify_foreach_event()`, 3006respectively. And update type name of callback accordingly. 3007 3008### bdev 3009 3010The format of the data returned by the get_bdevs_iostat RPC has changed to 3011make it easier to parse. It now returns an object with a "ticks" object 3012and "bdevs" array with the per-bdev statistics. 3013 3014A new bdev module `delay` has been added which simulates a drive latency when placed 3015on top of a Null bdev. This module is intended only for testing and can be created using 3016the new RPC `bdev_delay_create`. That RPC takes the name of the underlying bdev as well 3017as average and p99 latency arguments for both read and write operations. Average latency is 3018defined as a value close to what you would expect a perf tool such as FIO to report back as 3019the mean latency of all I/O submitted to the drive. p99 latency is defined as the value one 3020would expect the drive to see the slowest 1% of I/O report. For underlying drives with already 3021significant latency, the latency values provided to the drive will be additive. This should be 3022taken into account if trying to achieve an artificial latency on top of an nvme drive or aio device. 3023 3024DIF reference tag remapping is now supported for partition type virtual bdev 3025modules. When using partition type virtual bdevs, block address space is 3026remapped during I/O processing and DIF reference tag is remapped accordingly. 3027 3028Added spdk_bdev_*_with_md() functions allowing for IO with metadata being transferred in 3029separate buffer. To check support for separate metadata, use spdk_bdev_is_md_separate(). 3030 3031All bdevs now have a UUID. For devices whose backing hardware does not provide a UUID, 3032one is automatically generated. Across runs of SPDK, bdevs whose UUID is automatically 3033generated may change. 3034 3035A new virtual bdev module `compress` has been added to provide compression services on top of 3036a thinly provisioned logical volume. See documentation for complete details. 3037 3038### nvme 3039 3040Added an optional parameter `--io-queue-requests` to RPC `set_bdev_nvme_options`, which 3041can be used to change the number of requests allocated for one NVMe I/O queue. For 3042very big I/O size, e.g. 128MiB, with this option user will not get an error due to 3043limited requests in NVMe driver layer. 3044 3045Added spdk_nvme_ctrlr_get_transport_id() to get the transport ID from a 3046previously attached controller. 3047 3048Nvme Opal library spdk_opal_cmd deprecated. Adding separate command APIs. 3049NVMe Opal library add support for activating locking SP which will make the transaction 3050from "Manufactured-Inactive" state to "Manufactured" state. Upon successfully invoking 3051of this method, lock and unlock features will be enabled. 3052NVMe Opal library add support for locking/unlocking range and list locking range info. 3053NVMe opal library add support for multiuser. Admin can enable user and add user to specific 3054locking range and the user can lock/unlock his range. 3055 3056Added spdk_nvme_ctrlr_io_cmd_raw_no_payload_build() allowing a caller to pass 3057a completely formed command to an NVMe submission queue (buffer addresses and all). 3058This is supported on the PCIe transport only. 3059 3060Added spdk_nvme_get_ctrlr_registers() to return a pointer to the virtual address 3061of the NVMe controller registers. This is supported on the PCIe transport only. 3062 3063Added additional options to the spdk_nvme_ctrlr_alloc_qpair() option parameter 3064structure to allow caller to override the virtual and optionally physical address 3065of the submission and completion queue pair to be created. This is supported on 3066the PCIe transport only. 3067 3068Added `disable_error_logging` to struct spdk_nvme_ctrlr_opts, which disables 3069logging of failed requests. By default logging is enabled. 3070 3071Added spdk_nvme_qpair_print_command(), spdk_nvme_qpair_print_completion() and 3072spdk_nvme_cpl_get_status_string(). Allowing for easier display of error messages. 3073 3074Added support for NVMe Sanitize command. 3075 3076### env 3077 3078The parameter `free_space` has been added to spdk_ring_enqueue() to wait when 3079the ring is almost full and resume when there is enough space available in 3080the ring. 3081 3082A new API `spdk_mempool_lookup` has been added to lookup the memory pool created 3083by the primary process. 3084 3085Added spdk_pci_get_first_device() and spdk_pci_get_next_device() to allow 3086iterating over PCI devices detected by SPDK. Because of this, all SPDK APIs 3087to attach/detach PCI devices are no longer thread safe. They are now meant to 3088be called from only a single thread only, the same only that called spdk_env_init(). 3089This applies to the newly added APIs as well. 3090 3091### vpp 3092 3093SPDK now supports VPP version 19.04.2, up from VPP 18.01. 3094 3095VPP socket abstraction now uses VPP Session API, instead of VLC (VPP Communications Library). 3096This allows for better control over sessions and queues. 3097Please see VPP documentation for more details: 3098[VPP Host Stack](https://wiki.fd.io/view/VPP/HostStack) 3099 3100### sock 3101 3102Add spdk_sock_get_optimal_sock_group(), which returns the optimal sock group for 3103this socket. When a socket is created, it is often assigned to a sock group using 3104spdk_sock_group_add_sock so that a set of sockets can be polled more efficiently. 3105For some network devices, it is optimal to assign particular sockets to specific 3106sock groups. This API is intended to provide the user with that information. 3107 3108spdk_sock_group_get_ctx() was added to return the context of the spdk_sock_group. 3109spdk_sock_group_create() is updated to allow input the user provided ctx. 3110spdk_sock_set_priority() is added to set the priority of the socket. 3111 3112### rpc 3113 3114Added thread_get_stats RPC method to retrieve existing statistics. 3115 3116Added nvmf_get_stats RPC method to retrieve NVMf subsystem statistics. 3117 3118Response buffers for RPC requests are now always pre-allocated, which implies 3119that all spdk_jsonrpc_begin_result() calls always succeed and return a valid 3120buffer for JSON response. RPC calls no longer need to check if the buffer is 3121non-NULL. 3122 3123Added SPDK_RPC_REGISTER_ALIAS_DEPRECATED to help with deprecation process when 3124renaming existing RPC. First time a deprecated alias is used, it will print 3125a warning message. 3126 3127RPC `get_rpc_methods` was renamed `rpc_get_methods`. The old name is still usable, 3128but is now deprecated. 3129 3130### blobstore 3131 3132A snapshot can now be deleted if there is only a single clone on top of it. 3133 3134### build 3135 3136Preliminary support for cross compilation is now available. Targeting an older 3137CPU on the same architecture using your native compiler can be accomplished by 3138using the `--target-arch` option to `configure` as follows: 3139 3140~~~bash 3141./configure --target-arch=broadwell 3142~~~ 3143 3144Additionally, some support for cross-compiling to other architectures has been 3145added via the `--cross-prefix` argument to `configure`. To cross-compile, set CC 3146and CXX to the cross compilers, then run configure as follows: 3147 3148~~~bash 3149./configure --target-arch=aarm64 --cross-prefix=aarch64-linux-gnu 3150~~~ 3151 3152### vhost 3153 3154A security vulnerability has been identified and fixed in SPDK Vhost-SCSI target. 3155A malicious client (e.g. a virtual machine) could send a carefully prepared, 3156invalid I/O request to crash the entire SPDK process. All users of SPDK Vhost-SCSI 3157target are strongly recommended to update. All SPDK versions < 19.07 are affected. 3158 3159By default, SPDK will now rely on upstream DPDK's rte_vhost instead of its fork 3160located inside SPDK repo. The internal fork is still kept around to support older 3161DPDK versions, but is considered legacy and will be eventually removed. 3162 3163`configure` will now automatically use the upstream rte_vhost if the used DPDK 3164version is >= 19.05. 3165 3166spdk_vhost_init() is now asynchronous and accepts a completion callback. 3167 3168### iscsi target 3169 3170A security vulnerability has been identified and fixed in SPDK iSCSI target. 3171A malicious client (e.g. an iSCSI initiator) could send a carefully prepared, 3172invalid I/O request to crash the entire SPDK process. All users of SPDK iSCSI 3173target are strongly recommended to update. All SPDK versions < 19.07 are affected. 3174 3175### thread 3176 3177Exposed spdk_set_thread() in order for applications to associate 3178with SPDK thread when necessary. 3179 3180Added spdk_thread_destroy() to allow framework polling the thread to 3181release resources associated with that thread. 3182 3183## v19.04: Compression bdev, Notification library, NVMe Opal support 3184 3185### nvme 3186 3187Added asynchronous probe support. New APIs spdk_nvme_probe_async(), 3188spdk_nvme_connect_async() and spdk_nvme_probe_poll_async() were added to enable 3189this feature. spdk_nvme_probe_async() and spdk_nvme_connect_async() return a 3190context associated with the specified controllers. Users then call 3191spdk_nvme_probe_poll_async() until it returns 0, indicating that the operation 3192completed. 3193 3194A new qpair creation option, delay_pcie_doorbell, was added. This can be passed 3195to spdk_nvme_alloc_io_qpair(). This makes the I/O submission functions, such as 3196spdk_nvme_ns_writev(), skip ringing the submission queue doorbell. Instead the 3197doorbell will be rung as necessary inside spdk_nvme_qpair_process_completions(). 3198This can result in significantly fewer MMIO writes to the doorbell register 3199under heavy load, greatly improving performance. 3200 3201spdk_nvme_ctrlr_get_regs_cmbsz() was added to report the size of the controller 3202memory buffer, if available. 3203 3204spdk_nvme_ctrlr_get_flags() was added to return controller feature 3205flags. Two flags are currently tracked: 3206SPDK_NVME_CTRLR_SGL_SUPPORTED 3207SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED 3208 3209The NVMe hotplug poller is now able to detach devices hot-removed from the system 3210via `/sys/bus/pci/devices/<bdf>/remove` and `/sys/bus/pci/devices/<bdf>/driver/unbind`. 3211 3212Opal support was added for scan, take ownership, revert TPer, and dumping device 3213info. The nvme_manage tool can be used to perform these operations. The public 3214API functions are spdk_nvme_ctrlr_security_receive() and 3215spdk_nvme_ctrlr_security_send(). This module should be considered experimental 3216pending additional features and tests. 3217 3218The NVMe-oF initiator is now able to transfer IO whose size is more than 128KiB 3219if the corresponding NVMe-oF target allows. 3220 3221### raid 3222 3223Added new strip_size_kb rpc param on create to replace the more ambiguous 3224strip_size. The strip_size rpc param is deprecated. 3225 3226Changed the raid bdev product_name from "Pooled Device" to "Raid Volume" 3227 3228### thread 3229 3230Added spdk_thread_has_pollers() function to verify if there are any registered 3231pollers to be run on the thread. Added spdk_thread_is_idle() function to check 3232if there are any scheduled operations to be performed on the thread at given 3233time. 3234 3235spdk_thread_create() now takes an optional CPU affinity mask that is passed to 3236the scheduler. 3237 3238spdk_thread_lib_init() now takes an optional context size. For each thread 3239created, an additional region of memory of this size will be allocated. A 3240pointer to this region of memory can be obtained by calling 3241spdk_thread_get_ctx(). The inverse operation is also available via 3242spdk_thread_get_from_ctx(). 3243 3244spdk_thread_poll() now optionally accepts the current time, in ticks. This can 3245be used to avoid some calls to spdk_get_ticks() internally. 3246 3247spdk_app_start() now only accepts a single context argument. 3248 3249### bdev 3250 3251An new API `spdk_bdev_get_data_block_size` has been added to get size of data 3252block except for metadata. 3253 3254spdk_vbdev_register() has been deprecated. spdk_bdev_register() should be used 3255instead. 3256 3257A mechanism for acquiring and releasing data buffers from bdev modules, used 3258to perform zero copy operations, was added. 3259 3260New APIs spdk_bdev_get_md_size(), spdk_bdev_is_md_interleaved(), spdk_bdev_get_dif_type(), 3261spdk_bdev_is_dif_head_of_md(), and spdk_bdev_is_dif_check_enabled() have been 3262added to get metadata and DIF settings. 3263 3264Bdevs claimed by the `examine_config` callback will be now further examined in the 3265`examine_disk` callback. 3266 3267spdk_bdev_io_get_io_channel() was added as a convenient way to get an io_channel 3268from a bdev_io. 3269 3270### ocf 3271 3272Updated OCF submodule to OCF v19.3.2 3273 3274Added support for many-to-one configuration for OCF bdev. 3275Multiple core devices can now be cached on single cache device. 3276 3277Added persistent metadata support, allowing to restore cache state after shutdown. 3278During start of SPDK application, the devices are examined and if OCF metadata 3279is present - appropriate OCF bdevs will be recreated. 3280 3281Added Write-Back mode support. In this mode, data is first written to 3282caching device and periodically synchronized to the core devices. 3283Dirty data is saved as persistent metadata on cache device, 3284allowing for safe restore during application restart. 3285For more details please see OCF documentation: 3286[OpenCAS cache configuration](https://open-cas.github.io/cache_configuration.html) 3287 3288### NVMe-oF Target 3289 3290Support for per-device shared receive queues in the RDMA transport has been added. 3291It is enabled by default for any device that supports it. 3292 3293The size of a shared receive queue is defined by transport configuration file parameter 3294`MaxSRQDepth` and `nvmf_create_transport` RPC method parameter `max_srq_depth`. 3295Default size is 4096. 3296 3297Add model number as parameter to construct_nvmf_subsystem (-d option), 3298rather than using hardcoded define. 3299 3300DIF passthrough feature has been added. DIF setting of the allocated bdevs is 3301exposed to the NVMe-oF initiator and data with DIF from the NVMe-oF initiator is 3302passed through to the allocated bdevs. 3303 3304### env 3305 3306The `phys_addr` parameter in spdk_malloc() and spdk_zmalloc() has been deprecated. 3307For retrieving physical addresses, spdk_vtophys() should be used instead. 3308 3309spdk_realloc() has been added to reallocate DMA/shared memory. 3310 3311spdk_pci_device_is_removed() has been added to let the upper-layer SPDK drivers know 3312that device has a pending external hotremove request. 3313 3314spdk_env_fini() and spdk_env_dpdk_post_fini() were added to release any resources 3315allocated by spdk_env_init() or spdk_env_dpdk_post_init() respectively. It is expected 3316that common usage of those functions is to call them just before terminating the process. 3317 3318Added spdk_iommu_is_enabled() to report if SPDK application is using IOMMU for DMA. 3319 3320### DPDK 3321 3322Dropped support for DPDK 17.07 and earlier, which SPDK won't even compile with right now. 3323 3324Updated DPDK submodule to DPDK 19.02. 3325 3326### rpc 3327 3328New `get_spdk_version` RPC method is introduced to get version info of the running SPDK application. 3329 3330The `start_nbd_disk` RPC method now take nbd_device as an optional parameter. If nbd_device 3331is specified, use that specified nbd device. If it's not specified, pick available one. 3332 3333### iSCSI target 3334 3335DIF strip and insert is now supported. DIF settings are not exposed to the iSCSI initiator. 3336DIF is attached into data for write I/O and stripped from data for read I/O. 3337 3338### vhost 3339 3340Added experimental support for running with the external, upstream rte_vhost library. 3341This can be enabled by configuring SPDK with an `--without-internal-vhost-lib` flag. 3342The minimum supported rte_vhost version (DPDK version) is 19.05-rc1. 3343 3344As a result of fuzz testing, a lot of data races in vhost-scsi LUN hotplug path were identified and 3345fixed. Those data races could have potentially resulted in SPDK crashes, RPC hangs, or memory leaks 3346if Vhost-SCSI LUN hotplug RPCs were executed while connected VMs were in the middle of restarting. 3347 3348The SCSI target id in `add_vhost_scsi_lun` RPC is now optional. If `-1` is passed, the first 3349unoccupied target id will be used. 3350 3351### AIO 3352 3353AIO bdev module can now reap I/O completions directly from userspace, significantly improving 3354the overall performance. 3355 3356### blobfs 3357 3358Synchronous IO operations no longer use spdk_io_channel, but instead use 3359spdk_fs_thread_ctx. The behavior is otherwise identical. 3360 3361### OCF 3362 3363Added support for caching multiple bdevs using a single bdev as a cache. 3364 3365### notify 3366 3367Added the notify library that provides a high performance local event bus 3368between libraries. Example usage was added to bdev module, which reports 3369notifications for added and removed bdevs via RPC. 3370 3371### sock 3372 3373Added new API spdk_sock_readv() to the sock library for performing vectored 3374reads. 3375 3376### event 3377 3378The function spdk_subsystem_init() no longer requires spdk_event as an argument. 3379 3380Changed API of spdk_subsystem_config_json() to no longer be asynchronous. 3381 3382### io_uring 3383 3384A bdev module that issues I/O to kernel block devices using the new io_uring Linux kernel 3385API was added. This module requires liburing. 3386 3387### build 3388 3389Options to easily compile with profile guided optimization have been added to 3390`configure`. To use profile guided optimization with SPDK, run 3391`./configure --with-pgo-capture`, build SPDK, then run a workload of your 3392choosing. Then, simply run `./configure --with-pgo-enable` and recompile to 3393build using the generated profile data. Profile guided optimization can yield 3394very large performance improvements, especially on GCC 8 and clang 7. This may 3395be combined with link time optimization which has been available under the 3396`--enable-lto` configure option for several releases. 3397 3398### compression bdev/reduce library 3399 3400Added "reduce" block compression scheme based on using SSDs for storing 3401compressed blocks of storage and persistent memory for metadata. Please see 3402[compression](https://spdk.io/doc/bdev.html) for more details. 3403 3404## v19.01: NVMe-oF TCP/IP Transport, Open Channel SSD Flash Translation Layer, Caching bdev based on OCF, ISA-L Support, DIF/DIX library 3405 3406### ocf bdev 3407 3408New virtual bdev module based on [Open CAS Framework](https://open-cas.github.io/) has been added. 3409This module allows for the use of one bdev to act as a high performance cache in front of another bdev. 3410Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_cas) for more details. 3411Only write through mode is currently supported and this feature is considered experimental. 3412 3413### event framework 3414 3415For `spdk_app_parse_args`, add return value to the callback which parses application 3416specific command line parameters to protect SPDK applications from crashing by invalid 3417values from user input. 3418 3419By default, all SPDK applications will now reserve all hugepages at runtime. The pre-reserved 3420memory size can be still set with `-s` or `--mem-size` option, although the default value 3421was reduced down to 0. 3422 3423A custom hugetlbfs directory can now be specified via spdk_app_opts. 3424This can be used to configure hugepages with different sizes, a different size limit, 3425or different access permissions than the system's default hugepage pool. 3426SPDK applications can specify a custom hugetlbfs mount with the `--huge-dir` option. 3427 3428### environment 3429 3430spdk_vtophys() has been refactored to accept length of the translated region as a new 3431parameter. The function will now update that parameter with the largest possible value 3432for which the memory is contiguous in the physical memory address space. 3433 3434The following functions were removed: 3435 3436- spdk_pci_nvme_device_attach() 3437- spdk_pci_nvme_enumerate() 3438- spdk_pci_ioat_device_attach() 3439- spdk_pci_ioat_enumerate() 3440- spdk_pci_virtio_device_attach() 3441- spdk_pci_virtio_enumerate() 3442 3443They were replaced with generic spdk_pci_device_attach() and spdk_pci_enumerate() which 3444require a new spdk_pci_driver object to be provided. It can be one of the following: 3445 3446- spdk_pci_nvme_get_driver() 3447- spdk_pci_ioat_get_driver() 3448- spdk_pci_virtio_get_driver() 3449 3450spdk_pci_hook_device() and spdk_pci_unhook_device() were added. Those allow adding a virtual 3451spdk_pci_device into the SPDK PCI subsystem. A virtual device calls provided callbacks for 3452each BAR mapping request or PCI config access. It's attachable with spdk_pci_device_attach() 3453or spdk_pci_enumerate() like any other device. 3454 3455A new spdk_pause() function was added to pause CPU execution for an implementation specific 3456amount of time. Quoting from DPDK function this is based on: "This call is intended for 3457tight loops which poll a shared resource or wait for an event. A short pause within the loop 3458may reduce the power consumption." 3459 3460A new public header file env_dpdk.h has been introduced, and function spdk_env_dpdk_post_init 3461is added into it. If user is using DPDK, and already called rte_eal_init, then include 3462include/spdk/env_dpdk.h, and call spdk_env_dpdk_post_init() instead of spdk_env_init. 3463 3464ISA-L has been added as an SPDK submodule. ISA-L is enabled by default on x86 architecture 3465to accelerate algorithms such as CRC for iSCSI and NVMe-oF. Users may still disable ISA-L 3466by explicitly passing --without-isal to the configure script. 3467 3468### util 3469 3470A new uuid API `spdk_uuid_copy` was added to make a copy of the source uuid. 3471 3472An new parameter `init_crc` representing the initial CRC value was added to 3473`spdk_crc16_t10dif`. The parameter can be used to calculate a CRC value spanning 3474multiple separate buffers. 3475 3476New DIF APIs were added to generate and verify DIF by byte granularity for both DIF and DIX 3477formats. Among them, DIF with copy APIs will be usable to emulate DIF operations such as DIF 3478insert and strip. 3479 3480Added `spdk_strtol` and `spdk_strtoll` to provide additional error checking around `strtol` 3481and `strtoll`. 3482 3483Added `spdk_sprintf_append_realloc` and `spdk_vsprintf_append_realloc` for appending a string 3484with automatic buffer re-allocation. 3485 3486### nvme 3487 3488Wrapper functions spdk_nvme_ctrlr_security_send() and spdk_nvme_ctrlr_security_receive() are 3489introduced to support further security protocol development. 3490 3491admin_timeout_ms was added to NVMe controller initialization options, users 3492can change the default value when probing a controller. 3493 3494Add two new fields "header_digest" and "data_digest" in struct spdk_nvme_ctrlr_opts, 3495it will be used to enable the digest support for the NVMe/TCP transport. 3496 3497Add a new TCP/IP transport(located in lib/nvme/nvme_tcp.c) in nvme driver. With 3498this new transport, it can be used to connect the NVMe-oF target with the 3499same TCP/IP support. 3500 3501Added API, spdk_nvme_ctrlr_is_discovery(), to indicate whether the ctrlr 3502arg refers to a Discovery Controller or not. 3503 3504Added an API function `spdk_nvme_host_id_parse` and corresponding object `spdk_nvme_host_id` 3505for parsing host address and host service ID arguments on a per connection basis. 3506 3507The RPC `construct_nvme_bdev` now allows a user to specify a source address and service id for the host to 3508use when connecting to the controller backing the NVMe bdev. 3509 3510### NVMe-oF Target 3511 3512The `spdk_nvmf_tgt_opts` struct has been deprecated in favor of `spdk_nvmf_transport_opts`. 3513Users will no longer be able to specify target wide I/O parameters. `spdk_nvmf_tgt_listen` 3514will also no longer implicitly initialize a transport with the default target options (since 3515there are none). Instead, a user must manually instantiate the transport with `spdk_nvmf_transport_create` 3516prior to calling `spdk_nvmf_tgt_listen`. 3517 3518Related to the previous change, the rpc `set_nvmf_target_options` has been renamed to 3519`set_nvmf_target_max_subsystems` to indicate that this is the only target option available for the user to edit. 3520 3521Added fields `num_shared_buffers` and `buf_cache_size` in struct spdk_nvmf_transport_opts, 3522and also updated the related rpc function nvmf_create_transport, to make this 3523configurable parameter available to users. The `num_shared_buffers` is used to 3524configure the shared buffer numbers of the transport used by RDMA or TCP transport. 3525`buf_cache_size` configures number of shared buffers to cache per poll group. 3526 3527### nvmf 3528 3529Add a new TCP/IP transport (located in lib/nvmf/tcp.c). With this transport, 3530the SPDK NVMe-oF target can have a new transport, and can serve the NVMe-oF 3531protocol via TCP/IP from the host. 3532 3533Added optional mechanism to modify the RDMA transport's behavior when creating protection domains and registering memory. 3534By default, the RDMA transport will use the ibverbs library to create protection domains and register memory. 3535Using `spdk_nvme_rdma_init_hooks` will subvert that and use an existing registration. 3536 3537### bdev 3538 3539Added `enable_bdev_histogram` and `get_bdev_histogram` RPC commands to allow gathering latency data for specified bdev. 3540Please see [documentation](https://spdk.io/doc/bdev.html#rpc_bdev_histogram) for more details. 3541 3542Added `required_alignment` field to `spdk_bdev`, that specifies an alignment requirement for data buffers associated with an spdk_bdev_io. 3543Bdev layer will automatically double buffer any spdk_bdev_io that violates this alignment, before the spdk_bdev_io is submitted to the bdev module. 3544 3545On shutdown, bdev unregister now proceeds in top-down fashion, with 3546claimed bdevs skipped (these will be unregistered later, when virtual 3547bdev built on top of the respective base bdev unclaims it). This 3548allows virtual bdevs to be shut down cleanly as opposed to the 3549previous behavior that didn't differentiate between hotremove and 3550planned shutdown. 3551 3552The `delete_bdev` RPC is now deprecated. Users should instead use the specific deletion RPC 3553for the bdev type to be removed (i.e. delete_malloc_bdev). 3554 3555Added support for separate bandwidth rate limits for read and write to QoS in bdev layer. 3556 3557Bdev I/O statistics now track unmap operations. 3558 3559### logical volumes 3560 3561Logical volume bdev can now be marked as read only using `set_read_only_lvol_bdev` RPC. 3562This allows for basing clones on top of lvol_bdev without first creating a snapshot. 3563 3564Added option to change method for data erasure when deleting lvol or resizing down. 3565Default of unmapping clusters can now be changed to writing zeroes or no operation. 3566 3567Added option to change method for erasing data region on lvol store creation. 3568Default of unmapping can now be changed to writing zeroes or no operation. 3569 3570### log 3571 3572"trace flags" are now referred to as "log flags" in the SPDK log API. The 3573set_trace_flag, clear_trace_flag and get_trace_flags RPCs are now deprecated, 3574and set_log_flag, clear_log_flag and get_log_flags RPCs have been added. 3575 3576### trace 3577 3578New `trace_record` application was added. It can be used to poll spdk trace shm file and 3579append any new trace entries into another specified file. This can help retain those entries 3580that would otherwise be overwritten in the shm file. See 3581[Capturing sufficient trace events](https://spdk.io/doc/nvmf_tgt_tracepoints.html#capture_trace_events) 3582for more details. 3583 3584Number of trace entries in circular buffer per lcore can now be assigned by starting SPDK app 3585with argument "--num-trace-entries <NUM>" provided. 3586 3587New `get_tpoint_group_mask` RPC was added to get current tpoint_group_mask, and 3588each tpoint group status. 3589New `enable_tpoint_group` and `disable_tpoint_group` RPC were added to enable or 3590disable a specific tpoint group. 3591 3592### ftl 3593 3594EXPERIMENTAL: Added basic flash translation layer module allowing for using Open Channel SSDs as 3595block devices. The module is split into the library (located in lib/ftl) and bdev_ftl 3596(lib/bdev/ftl). See the [documentation](https://spdk.io/doc/ftl.html) for more details. 3597 3598### vhost 3599 3600A security vulnerability has been identified and fixed in the SPDK vhost target. A malicious 3601vhost client (i.e. virtual machine) could carefully construct a circular descriptor chain which 3602would result in a partial denial of service in the SPDK vhost target. These types of descriptor 3603chains are now properly detected by the vhost target. All SPDK vhost users serving untrusted 3604vhost clients are strongly recommended to upgrade. (Reported by Dima Stepanov and Evgeny 3605Yakovlev.) 3606 3607Vhost SCSI and Vhost Block devices can now accept multiple connections on the same socket file. 3608Each connection (internally called a vhost session) will have access to the same storage, but 3609will use different virtqueues, different features and possibly different memory. 3610 3611### vhost scsi 3612 3613SCSI target hotremove can now be performed even without the VIRTIO_SCSI_F_HOTPLUG feature negotiated. 3614Regardless of VIRTIO_SCSI_F_HOTPLUG support, the hotremoval will be still reported through SCSI sense codes. 3615 3616### DPDK 3617 3618DPDK submodule was updated to DPDK 18.11. Note that SPDK does not fully leverage the new 3619multi-process device hotplug yet and continues to work the same way it always did. 3620 3621Dropped support for DPDK 16.07 and earlier, which SPDK won't even compile with right now. 3622 3623### RPC 3624 3625The following RPC commands deprecated in the previous release are now removed: 3626 3627- construct_virtio_user_scsi_bdev 3628- construct_virtio_pci_scsi_bdev 3629- construct_virtio_user_blk_bdev 3630- construct_virtio_pci_blk_bdev 3631- remove_virtio_scsi_bdev 3632- construct_nvmf_subsystem 3633 3634### Miscellaneous 3635 3636The configure options `--with-raid` and `--without-raid` that were deprecated in the previous 3637release are now removed. 3638 3639### nbd 3640 3641Starting nbd using `spdk_nbd_start` is now performed asynchronously. 3642 3643### net framework 3644 3645Net framework initialization and finish is now done asynchronously. 3646 3647### rpc 3648 3649Added `spdk_rpc_is_method_allowed` function for checking whether method is permitted in a given state. 3650Added `spdk_rpc_get_state` to check current state of RPC server. 3651RPC `wait_subsystem_init` has been added to allow clients to block until all subsystems are initialized. 3652 3653### json rpc 3654 3655JSON RPC client is now running in non-blocking mode. Requests are sent and received during spdk_jsonrpc_client_poll. 3656JSON RPC server can now receive a callback on connection termination or server shutdown using `spdk_jsonrpc_conn_add_close_cb` 3657and `spdk_jsonrpc_conn_del_close_cb`. 3658 3659## v18.10: Dynamic memory allocation, Crypto Virtual Bdev, jsonrpc-client, SPDKCLI iSCSI and NVMe-oF support 3660 3661### nvme 3662 3663spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive() 3664were added to support sending or receiving security protocol data to or from 3665nvme controller. 3666 3667spdk_nvme_ns_get_extended_sector_size() was added. This function includes 3668the metadata size per sector (if any). spdk_nvme_ns_get_sector_size() still 3669returns only the data size per sector, not including metadata. 3670 3671New `send_nvme_cmd` RPC was added to allow sending NVMe commands directly to NVMe controller. 3672See the [send_nvme_cmd](http://spdk.io/doc/jsonrpc.html#rpc_send_nvme_cmd) documentation 3673for more details. 3674 3675### Build System 3676 3677New `configure` options, `--with-shared` and `--without-shared` 3678[default], provide the capability to build, or not, SPDK shared libraries. 3679This includes the single SPDK shared lib encompassing all of the SPDK 3680static libs as well as individual SPDK shared libs corresponding to 3681each of the SPDK static ones. Although the production of the shared 3682libs conforms with conventional version naming practices, such naming 3683does not at this time confer any SPDK ABI compatibility claims. 3684 3685### bdev 3686 3687spdk_bdev_alias_del_all() was added to delete all alias from block device. 3688 3689A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev 3690Framework. The module initially uses a software AESNI CBC cipher with experimental support for the 3691Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work 3692may include additional ciphers as well as consideration for authentication. 3693 3694The RAID virtual bdev module is now always enabled by default. The configure --with-raid and 3695--without-raid options are now ignored and deprecated and will be removed in the next release. 3696 3697Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer. 3698See the new [set_bdev_qos_limit](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit) 3699documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at 370018.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both 3701bandwidth and IOPS limits. 3702 3703spdk_bdev_config_json() and corresponding `get_bdevs_config` RPC was removed. 3704 3705### Environment Abstraction Layer and Event Framework 3706 3707The size parameter of spdk_mem_map_translate is now a pointer. This allows the 3708function to report back the actual size of the translation relative to the original 3709request made by the user. 3710 3711A new structure spdk_mem_map_ops has been introduced to hold memory map related 3712callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc 3713in lieu of the notify callback. 3714 3715### DPDK 18.08 3716 3717The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically 3718utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1. 3719 3720Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup, 3721but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try 3722to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using 3723only as many hugepages as it is really needed. 3724 3725Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed 3726to be physically contiguous. 3727 3728### I/OAT 3729 3730I/OAT driver can now reinitialize I/OAT channels after encountering DMA errors. 3731 3732### iscsi target 3733 3734Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC 3735method for CHAP authentication in discovery sessions have been changed to 3736align with `construct_target_node` RPC method. Old names are still usable 3737but will be removed in future release. 3738 3739`set_iscsi_discovery_auth` and `set_iscsi_target_node_auth` RPC methods have 3740been added to set CHAP authentication for discovery sessions and existing 3741target nodes, respectively. 3742 3743The SPDK iSCSI target supports an AuthFile which can be used to load CHAP 3744shared secrets when the iSCSI target starts. SPDK previously provided a 3745default location for this file (`/usr/local/etc/spdk/auth.conf`) if none was 3746specified. This default has been removed. Users must now explicitly specify 3747the location of this file to load CHAP shared secrets from a file, or use 3748the related iSCSI RPC methods to add them at runtime. 3749 3750### iscsi initiator 3751 3752The SPDK iSCSI initiator is no longer considered experimental and becomes 3753a first-class citizen among bdev modules. The basic usage has been briefly 3754described in the bdev user guide: [iSCSI bdev](https://spdk.io/doc/bdev.html#bdev_config_iscsi) 3755 3756### Miscellaneous 3757 3758The SPDK application framework can now parse long name command line parameters. 3759Most single-character parameters have a long name equivalent now. See the 3760[Command Line Parameters](https://spdk.io/doc/app_overview.html) documentation 3761for details or use the `--help` command line parameter to list all available 3762params. 3763 3764bdevperf `-s` param (io size) was renamed to `-o` as `-s` had been already 3765used by existing apps for memory size. 3766 3767bdevio can now accept all SPDK command line parameters. The config now has to 3768be provided with `-c` or `--config` param. 3769 3770The following ioat/perf and nvme/perf parameters were renamed as well: 3771 `-s` (io size) to `-o` 3772 `-d` (mem size) to `-s` 3773 3774The ReactorMask config file parameter has been deprecated. Users should 3775use the -m or --cpumask command line option to specify the CPU core mask 3776for the application. 3777 3778Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt 3779and vhost. Config file pathnames may now only be specified using the 3780-c command line option. 3781 3782Users may no longer set DPDK_DIR in their environment to specify the 3783location of the DPDK installation used to build SPDK. Using DPDK_DIR 3784has not been the documented nor recommended way to specify the DPDK 3785location for several releases, but removing it ensures no unexpected 3786surprises for users who may have DPDK_DIR defined for other reasons. 3787Users should just use the "configure" script to specify the DPDK 3788location before building SPDK. 3789 3790Although we know that many developers still use Python 2 we are officially 3791switching to Python3 with requirement that all new code must be valid also 3792for Python 2 up to the EOL which is year 2020. 3793 3794Invoking interpreter explicitly is forbidden for executable scripts. There 3795is no need to use syntax like "python ./scripts/rpc.py". All executable 3796scripts must contain proper shebang pointing to the right interpreter. 3797Scripts without shebang mustn't be executable. 3798 3799A Python script has been added to enable conversion of old INI config file 3800to new JSON-RPC config file format. This script can be found at 3801scripts/config_converter.py. Example how this script can be used: 3802~~~{.sh} 3803cat old_format.ini | scripts/config_converter.py > new_json_format.json 3804~~~ 3805 3806### Sock 3807 3808Two additional parameters were added to spdk_sock_get_addr() for the server 3809port and client port. These parameters are named "sport" and "cport" 3810respectively. 3811 3812### Virtio 3813 3814The following RPC commands have been deprecated: 3815 3816- construct_virtio_user_scsi_bdev 3817- construct_virtio_pci_scsi_bdev 3818- construct_virtio_user_blk_bdev 3819- construct_virtio_pci_blk_bdev 3820- remove_virtio_scsi_bdev 3821 3822The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev` 3823command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev` 3824was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI 3825devices. 3826 3827### Blobfs 3828 3829spdk_file_get_id() returning unique ID for the file was added. 3830 3831### JSON 3832 3833Added jsonrpc-client C library intended for issuing RPC commands from applications. 3834 3835Added API enabling iteration over JSON object: 3836 3837- spdk_json_find() 3838- spdk_json_find_string() 3839- spdk_json_find_array() 3840- spdk_json_object_first() 3841- spdk_json_array_first() 3842- spdk_json_next() 3843 3844### Blobstore 3845 3846Blobstore I/O operations are now based on io_units, instead of blobstore page size. 3847The io_unit size is now the same as the underlying block device's block size. 3848Logical volumes built on a block device with 512B block size can now be used as boot devices 3849in QEMU. 3850 3851### SPDKCLI 3852 3853The SPDKCLI interactive command tool for managing SPDK is no longer considered experimental. 3854Support for the iSCSI and NVMe-oF targets has been added. 3855 3856## v18.07: Raid, Infrastructure Improvements, Bug Fixes 3857 3858### bdev 3859 3860A new public header file bdev_module.h has been introduced to facilitate the 3861development of new bdev modules. This header includes an interface for the 3862spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of 3863multiple virtual bdevs on top of a single base bdev and should act as the 3864primary API for module authors. 3865 3866spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide 3867options. 3868 3869A mechanism for handling out of memory condition errors (ENOMEM) returned from 3870I/O submission requests at the bdev layer has been added. See 3871spdk_bdev_queue_io_wait(). 3872 3873The spdk_bdev_get_io_stat() function now returns cumulative totals instead of 3874resetting on each call. This allows multiple callers to query I/O statistics 3875without conflicting with each other. Existing users will need to adjust their 3876code to record the previous I/O statistics to calculate the delta between calls. 3877 3878I/O queue depth tracking and samples options have been added. See 3879spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and 3880spdk_bdev_set_qd_sampling_period(). 3881 3882### RAID module 3883 3884A new bdev module called "raid" has been added as experimental module which 3885aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note 3886that vhost will not work with this module because it does not yet have support 3887for multi-element io vectors. 3888 3889### Log 3890 3891The debug log component flag available on several SPDK applications has been 3892renamed from `-t` to `-L` to prevent confusion with tracepoints and to allow the 3893option to be added to tools that already use `-t` to mean something else. 3894 3895### Blobstore 3896 3897A new function, spdk_bs_dump(), has been added that dumps all of the contents of 3898a blobstore to a file pointer. This includes the metadata and is very useful for 3899debugging. 3900 3901Two new operations have been added for thin-provisioned blobs. 3902spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions 3903of the blob and populate them with the correct data by reading from the backing 3904blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate 3905clusters that correspond to data in the blob's immediate parent. Clusters 3906allocated to grandparents or that aren't allocated at all will remain 3907thin-provisioned. 3908 3909### BlobFS 3910 3911Changed the return type of spdk_file_truncate() from void to int to allow the 3912propagation of `ENOMEM` errors. 3913 3914### NVMe Driver 3915 3916The new API functions spdk_nvme_qpair_add_cmd_error_injection() and 3917spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error 3918emulation. Users can set a specified command to fail with a particular error 3919status. 3920 3921Changed the name `timeout_sec` parameter to `timeout_us` in 3922spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from 3923uint32_t to uint64_t. This will give users more fine-grained control over the 3924timeout period. 3925 3926Basic support for Open Channel SSDs was added. See nvme_ocssd.h 3927 3928### NVMe Over Fabrics 3929 3930The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback 3931as a parameter. 3932 3933spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs. 3934 3935spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed 3936number of namespaces for a given subsystem. 3937 3938### Build System 3939 3940The build system now generates a combined shared library (libspdk.so) that may 3941be used in place of the individual static libraries (libspdk_*.a). The combined 3942library includes all components of SPDK and is intended to make linking against 3943SPDK easier. The static libraries are also still provided for users that prefer 3944to link only the minimal set of components required. 3945 3946### git pre-commit and pre-push hooks 3947 3948The pre-commit hook will run `scripts/check_format.sh` and verify there are no 3949formatting errors before allowing `git commit` to run. The pre-push hook runs 3950`make CONFIG_WERROR=y` with and without `CONFIG_DEBUG=y` using both the gcc and 3951clang compiler before allowing `git push` to run. Following each DEBUG build 3952`test/unit/unittest.sh` is run and verified. Results are recorded in the 3953`make.log` file. 3954 3955To enable type: 'git config core.hooksPath .githooks'. To override after 3956configuration use the `git --no-verify` flag. 3957 3958### RPC 3959 3960The `start_nbd_disk` RPC method now returns the path to the kernel NBD device node 3961rather than always returning `true`. 3962 3963### DPDK 18.05 3964 3965The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports 3966dynamic memory allocation, but due to some issues found after the DPDK 18.05 release, 3967that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use 3968the legacy memory allocation model. The plan is to enable dynamic memory allocation 3969after the DPDK 18.08 release which should fix these issues. 3970 3971### Environment Abstraction Layer and Event Framework 3972 3973The spdk_mem_map_translate() function now takes a size parameter to indicate the size of 3974the memory region. This can be used by environment implementations to validate the 3975requested translation. 3976 3977The I/O Channel implementation has been moved to its own library - lib/thread. The 3978public API that was previously in spdk/io_channel.h is now in spdk/thread.h The 3979file spdk/io_channel.h remains and includes spdk/thread.h. 3980 3981spdk_reactor_get_tsc_stats was added to return interesting statistics for each 3982reactor. 3983 3984### IOAT 3985 3986IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable 3987option with "Yes" in `[Ioat]` section of the configuration file. The Disable option is 3988now deprecated and will be removed in a future release. 3989 3990## v18.04: Logical Volume Snapshot/Clone, iSCSI Initiator, Bdev QoS, VPP Userspace TCP/IP 3991 3992### vhost 3993 3994The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the 3995DPDK rte_vhost vulnerability [CVE-2018-1059](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1059). 3996Please see this [security advisory](https://access.redhat.com/security/cve/cve-2018-1059) 3997for additional information on the DPDK vulnerability. 3998 3999Workarounds have been added to ensure vhost compatibility with QEMU 2.12. 4000 4001EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the 4002[vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 4003 4004### Unified Target Application 4005 4006A new unified SPDK target application, `spdk_tgt`, has been added. This application combines the 4007functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target, 4008and vhost target. The new application can be managed through the existing configuration file and 4009[JSON-RPC](http://www.spdk.io/doc/jsonrpc.html) methods. 4010 4011### Env 4012 4013spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk(). 4014 4015New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added. 4016These new functions have a `flags` parameter that allows the user to specify whether the allocated 4017memory needs to be suitable for DMA and whether it should be shared across processes with the same 4018shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will 4019eventually be deprecated and removed. 4020 4021### Bdev 4022 4023A new optional bdev module interface function, `init_complete`, has been added to notify bdev modules 4024when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require 4025notification that the set of initialization examine() calls is complete. 4026 4027The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with 4028the spdk_bdev_get_uuid() function. 4029 4030Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the 4031[set_bdev_qos_limit_iops](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit_iops) documentation 4032for more details. 4033 4034### RPC 4035 4036The `[Rpc]` configuration file section, which was deprecated in v18.01, has been removed. 4037Users should switch to the `-r` command-line parameter instead. 4038 4039The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as 4040needed; previously, the response was limited to 32 kilobytes. 4041 4042### SPDKCLI 4043 4044EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available. 4045See the [SPDKCLI](http://www.spdk.io/doc/spdkcli.html) documentation for more details. 4046 4047### NVMe Driver 4048 4049EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is 4050experimental pending a functional allocator to free and reallocate CMB buffers. 4051 4052spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available. 4053 4054New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns() 4055have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if 4056a namespace ID is active. 4057 4058### NVMe-oF Target 4059 4060Namespaces may now be assigned unique identifiers via new optional `eui64` and `nguid` parameters 4061to the `nvmf_subsystem_add_ns` RPC method. Additionally, the NVMe-oF target automatically exposes 4062the backing bdev's UUID as the namespace UUID when available. 4063 4064spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion. 4065 4066### Blobstore 4067 4068A number of functions have been renamed: 4069 4070- spdk_bs_io_write_blob() => spdk_blob_io_write() 4071- spdk_bs_io_read_blob() => spdk_blob_io_read() 4072- spdk_bs_io_writev_blob() => spdk_blob_io_writev() 4073- spdk_bs_io_readv_blob() => spdk_blob_io_readv() 4074- spdk_bs_io_unmap_blob() => spdk_blob_io_unmap() 4075- spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes() 4076 4077The old names still exist but are deprecated. They will be removed in the v18.07 release. 4078 4079spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O 4080are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still 4081required to sync the updated metadata to disk. 4082 4083### Logical Volumes 4084 4085A new `destroy_lvol_bdev` RPC method to delete logical volumes has been added. 4086 4087Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination. 4088 4089New Snapshot and Clone functionalities have been added. User may create Snapshots of existing Lvols 4090and Clones of existing Snapshots. 4091See the [lvol snapshots](http://www.spdk.io/doc/logical_volumes.html#lvol_snapshots) documentation 4092for more details. 4093 4094Resizing logical volumes is now supported via the `resize_lvol_bdev` RPC method. 4095 4096### Lib 4097 4098A set of changes were made in the SPDK's lib code altering 4099instances of calls to `exit()` and `abort()` to return a failure instead 4100wherever reasonably possible. 4101 4102spdk_app_start() no longer exit()'s on an internal failure, but 4103instead returns a non-zero error status. 4104 4105spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid 4106option, but instead returns SPDK_APP_PARSE_ARGS_HELP and 4107SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS 4108on success. 4109 4110spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07. 4111 4112### I/O Channels 4113 4114The prototype for spdk_poller_fn() has been modified; it now returns a value indicating 4115whether or not the poller did any work. Existing pollers will need to be updated to 4116return a value. 4117 4118### iSCSI Target 4119 4120The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace 4121TCP/IP stack. See the [iSCSI VPP documentation](http://www.spdk.io/doc/iscsi.html#vpp) for more 4122details. 4123 4124### iSCSI initiator 4125 4126An iSCSI initiator bdev module has been added to SPDK. This module should be considered 4127experimental pending additional features and tests. More details can be found in 4128lib/bdev/iscsi/README. 4129 4130### PMDK 4131 4132The persistent memory (PMDK) bdev module is now enabled using `--with-pmdk` instead of 4133`--with-nvml`. This reflects the renaming of the persistent memory library from NVML to 4134PMDK. 4135 4136### Virtio Block driver 4137 4138A userspace driver for Virtio Block devices has been added. It was built on top of the 4139[Virtio](http://www.spdk.io/doc/virtio.html) library and can be managed similarly to 4140the Virtio SCSI driver. See the 4141[Virtio Block](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_blk) reference for 4142more information. 4143 4144### Virtio with 2MB hugepages 4145 4146The previous 1GB hugepage limitation has now been lifted. A new `-g` command-line option 4147enables SPDK Virtio to work with 2MB hugepages. 4148See [2MB hugepages](http://www.spdk.io/doc/virtio.html#virtio_2mb) for details. 4149 4150## v18.01: Blobstore Thin Provisioning 4151 4152### Build System 4153 4154The build system now includes a `make install` rule, including support for the common 4155`DESTDIR` and `prefix` variables as used in other build systems. Additionally, the prefix 4156may be set via the configure `--prefix` option. Example: `make install prefix=/usr`. 4157 4158### RPC 4159 4160A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/tmp/spdk.sock. 4161A -r option command line option has been added to enable an alternative UNIX domain socket location, 4162or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file 4163section is now deprecated and will be removed in the v18.04 release. 4164 4165### I/O Channels 4166 4167spdk_poller_register() and spdk_poller_unregister() were moved from the event 4168framework (include/spdk/event.h) to the I/O channel library 4169(include/spdk/io_channel.h). This allows code that doesn't depend on the event 4170framework to request registration and unregistration of pollers. 4171 4172spdk_for_each_channel() now allows asynchronous operations during iteration. 4173Instead of immediately continuing the interaction upon returning from the iteration 4174callback, the user must call spdk_for_each_channel_continue() to resume iteration. 4175 4176### Block Device Abstraction Layer (bdev) 4177 4178The poller abstraction was removed from the bdev layer. There is now a general purpose 4179abstraction for pollers available in include/spdk/io_channel.h 4180 4181### Lib 4182 4183A set of changes were made in the SPDK's lib code altering, 4184instances of calls to `exit()` and `abort()` to return a failure instead 4185wherever reasonably possible. This has resulted in return type changes of 4186the API for: 4187 4188- spdk_env_init() from type `void` to `int`. 4189- spdk_mem_map_init() from type `void` to `int`. 4190 4191Applications making use of these APIs should be modified to check for 4192a non-zero return value instead of relying on them to fail without return. 4193 4194### NVMe Driver 4195 4196SPDK now supports hotplug for vfio-attached devices. But there is one thing keep in mind: 4197Only physical removal events are supported; removing devices via the sysfs `remove` file will not work. 4198 4199### NVMe-oF Target 4200 4201Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available 4202cores round-robin. The "Core" option in the configuration file has been removed. 4203 4204### Blobstore 4205 4206A number of functions have been renamed: 4207 4208- spdk_bs_md_resize_blob() => spdk_blob_resize() 4209- spdk_bs_md_sync_blob() => spdk_blob_sync_md() 4210- spdk_bs_md_close_blob() => spdk_blob_close() 4211- spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names() 4212- spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value() 4213- spdk_blob_md_set_xattr() => spdk_blob_set_xattr() 4214- spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr() 4215- spdk_bs_md_create_blob() => spdk_bs_create_blob() 4216- spdk_bs_md_open_blob() => spdk_bs_open_blob() 4217- spdk_bs_md_delete_blob() => spdk_bs_delete_blob() 4218- spdk_bs_md_iter_first() => spdk_bs_iter_first() 4219- spdk_bs_md_iter_next() => spdk_bs_iter_next() 4220 4221The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument 4222rather than struct spdk_blob **. 4223 4224The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument 4225rather than struct spdk_blob **. 4226 4227Thin provisioning support has been added to the blobstore. It can be enabled by setting the 4228`thin_provision` flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext(). 4229 4230### NBD device 4231 4232The NBD application (test/lib/bdev/nbd) has been removed; Same functionality can now be 4233achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method. 4234See the [GPT](http://www.spdk.io/doc/bdev.html#bdev_config_gpt) documentation for more details. 4235 4236### FIO plugin 4237 4238SPDK `fio_plugin` now supports FIO 3.3. The support for previous FIO 2.21 has been dropped, 4239although it still remains to work for now. The new FIO contains huge amount of bugfixes and 4240it's recommended to do an update. 4241 4242### Virtio library 4243 4244Previously a part of the bdev_virtio module, now a separate library. Virtio is now available 4245via `spdk_internal/virtio.h` file. This is an internal interface to be used when implementing 4246new Virtio backends, namely Virtio-BLK. 4247 4248### iSCSI 4249 4250The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated 4251to an epoll/kqueue descriptor on the master core when idle. 4252 4253## v17.10: Logical Volumes 4254 4255### New dependencies 4256 4257libuuid was added as new dependency for logical volumes. 4258 4259libnuma is now required unconditionally now that the DPDK submodule has been updated to DPDK 17.08. 4260 4261### Block Device Abstraction Layer (bdev) 4262 4263An [fio](http://github.com/axboe/fio) plugin was added that can route 4264I/O to the bdev layer. See the [plugin documentation](https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin/) 4265for more information. 4266 4267spdk_bdev_unmap() was modified to take an offset and a length in bytes as 4268arguments instead of requiring the user to provide an array of SCSI 4269unmap descriptors. This limits unmaps to a single contiguous range. 4270 4271spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out. 4272If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using 4273write commands. 4274 4275New API functions that accept I/O parameters in units of blocks instead of bytes 4276have been added: 4277 4278- spdk_bdev_read_blocks(), spdk_bdev_readv_blocks() 4279- spdk_bdev_write_blocks(), spdk_bdev_writev_blocks() 4280- spdk_bdev_write_zeroes_blocks() 4281- spdk_bdev_unmap_blocks() 4282 4283The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be 4284retried later. 4285 4286### Linux AIO bdev 4287 4288The AIO bdev now allows the user to override the auto-detected block size. 4289 4290### NVMe driver 4291 4292The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field. 4293NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take 4294into account when splitting I/O requests. 4295 4296The HotplugEnable option in `[Nvme]` sections of the configuration file is now 4297"No" by default. It was previously "Yes". 4298 4299The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the 4300NVMe Controller associated with a given namespace. 4301 4302The NVMe library now allows the user to specify a host identifier when attaching 4303to a controller. The host identifier is used as part of the Reservations feature, 4304as well as in the NVMe-oF Connect command. The default host ID is also now a 4305randomly-generated UUID, and the default host NQN uses the host ID to generate 4306a UUID-based NQN. 4307 4308spdk_nvme_connect() was added to allow the user to connect directly to a single 4309NVMe or NVMe-oF controller. 4310 4311### NVMe-oF Target (nvmf_tgt) 4312 4313The NVMe-oF target no longer requires any in-capsule data buffers to run, and 4314the feature is now entirely optional. Previously, at least 4 KiB in-capsule 4315data buffers were required. 4316 4317NVMe-oF subsystems have a new configuration option, AllowAnyHost, to control 4318whether the host NQN whitelist is enforced when accepting new connections. 4319If no Host options have been specified and AllowAnyHost is disabled, the 4320connection will be denied; this is a behavior change from previous releases, 4321which allowed any host NQN to connect if the Host list was empty. 4322AllowAnyHost is disabled by default. 4323 4324NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number 4325of namespaces per subsystem is no longer limited. 4326 4327The NVMe-oF target now supports the Write Zeroes command. 4328 4329### Environment Abstraction Layer 4330 4331A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide 4332additional clarity when constructing spdk_mempools. Previously, -1 could be 4333passed and the library would choose a reasonable default, but this new value 4334makes it explicit that the default is being used. 4335 4336### Blobstore 4337 4338The blobstore super block now contains a bstype field to identify the type of the blobstore. 4339Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load(). 4340 4341spdk_bs_destroy() was added to allow destroying blobstore on device 4342with an initialized blobstore. 4343 4344spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable 4345scattered payloads. 4346 4347A CLI tool for blobstore has been added, allowing basic operations through either command 4348line or shell interface. See the [blobcli](https://github.com/spdk/spdk/tree/master/examples/blob/cli) 4349documentation for more details. 4350 4351### Event Framework 4352 4353The ability to set a thread name, previously only used by the reactor code, is 4354now part of the spdk_thread_allocate() API. Users may specify a thread name 4355which will show up in tools like `gdb`. 4356 4357### Log 4358 4359The spdk_trace_dump() function now takes a new parameter to allow the caller to 4360specify an output file handle (stdout or stderr, for example). 4361 4362### Logical Volumes 4363 4364Logical volumes library built on top of SPDK blobstore has been added. 4365It is possible to create logical volumes on top of other devices using RPC. 4366 4367See the [logical volumes](http://www.spdk.io/doc/logical_volumes.html) documentation for more information. 4368 4369### Persistent Memory 4370 4371A new persistent memory bdev type has been added. 4372The persistent memory block device is built on top of [libpmemblk](http://pmem.io/nvml/libpmemblk/). 4373It is possible to create pmem devices on top of pmem pool files using RPC. 4374 4375See the [Pmem Block Device](http://www.spdk.io/doc/bdev.html#bdev_config_pmem) documentation for more information. 4376 4377### Virtio SCSI driver 4378 4379A userspace driver for Virtio SCSI devices has been added. 4380The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application. 4381 4382See the [Virtio SCSI](http://www.spdk.io/doc/virtio.html) documentation and 4383[Getting Started](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_scsi) guide for more information. 4384 4385### Vhost target 4386 4387The vhost target application now supports live migration between QEMU instances. 4388 4389## v17.07: Build system improvements, userspace vhost-blk target, and GPT bdev 4390 4391### Build System 4392 4393A `configure` script has been added to simplify the build configuration process. 4394The existing CONFIG file and `make CONFIG_...` options are also still supported. 4395Run `./configure --help` for information about available configuration options. 4396 4397A DPDK submodule has been added to make building SPDK easier. If no `--with-dpdk` 4398option is specified to configure, the SPDK build system will automatically build a 4399known-good configuration of DPDK with the minimal options enabled. See the Building 4400section of README.md for more information. 4401 4402A [Vagrant](https://www.vagrantup.com/) setup has been added to make it easier to 4403develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant 4404section of README.md for more information. 4405 4406### Userspace vhost-blk target 4407 4408The vhost library and example app have been updated to support the vhost-blk 4409protocol in addition to the existing vhost-scsi protocol. 4410See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 4411 4412### Block device abstraction layer (bdev) 4413 4414A GPT virtual block device has been added, which automatically exposes GPT partitions 4415with a special SPDK-specific partition type as bdevs. 4416See the [GPT bdev documentation](http://www.spdk.io/doc/bdev.md#bdev_config_gpt) for 4417more information. 4418 4419### NVMe driver 4420 4421The NVMe driver has been updated to support recent Intel SSDs, including the Intel® 4422Optane™ SSD DC P4800X series. 4423 4424A workaround has been added for devices that failed to recognize register writes 4425during controller reset. 4426 4427The NVMe driver now allocates request tracking objects on a per-queue basis. The 4428number of requests allowed on an I/O queue may be set during `spdk_nvme_probe()` by 4429modifying `io_queue_requests` in the opts structure. 4430 4431The SPDK NVMe `fio_plugin` has been updated to support multiple threads (`numjobs`). 4432 4433spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override 4434controller-level options for each individual I/O queue pair. 4435Existing callers with qprio == 0 can be updated to: 4436~~~ 4437... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0); 4438~~~ 4439Callers that need to specify a non-default qprio should be updated to: 4440~~~ 4441struct spdk_nvme_io_qpair_opts opts; 4442spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts)); 4443opts.qprio = SPDK_NVME_QPRIO_...; 4444... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts)); 4445~~~ 4446 4447### Environment Abstraction Layer 4448 4449The environment abstraction layer has been updated to include several new functions 4450in order to wrap additional DPDK functionality. See `include/spdk/env.h` for the 4451current set of functions. 4452 4453### SPDK Performance Analysis with Intel® VTune™ Amplifier 4454 4455Support for SPDK performance analysis has been added to Intel® VTune™ Amplifier 2018. 4456 4457This analysis provides: 4458 4459- I/O performance monitoring (calculating standard I/O metrics like IOPS, throughput, etc.) 4460- Tuning insights on the interplay of I/O and compute devices by estimating how many cores 4461 would be reasonable to provide for SPDK to keep up with a current storage workload. 4462 4463See the VTune Amplifier documentation for more information. 4464 4465## v17.03: Blobstore and userspace vhost-scsi target 4466 4467### Blobstore and BlobFS 4468 4469The blobstore is a persistent, power-fail safe block allocator designed to be 4470used as the local storage system backing a higher-level storage service. 4471See the [blobstore documentation](http://www.spdk.io/doc/blob.html) for more details. 4472 4473BlobFS adds basic filesystem functionality like filenames on top of the blobstore. 4474This release also includes a RocksDB Env implementation using BlobFS in place of the 4475kernel filesystem. 4476See the [BlobFS documentation](http://www.spdk.io/doc/blobfs.html) for more details. 4477 4478### Userspace vhost-scsi target 4479 4480A userspace implementation of the QEMU vhost-scsi protocol has been added. 4481The vhost target is capable of exporting SPDK bdevs to QEMU-based VMs as virtio devices. 4482See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 4483 4484### Event framework 4485 4486The overhead of the main reactor event loop was reduced by optimizing the number of 4487calls to spdk_get_ticks() per iteration. 4488 4489### NVMe library 4490 4491The NVMe library will now automatically split readv/writev requests with scatter-gather 4492lists that do not map to valid PRP lists when the NVMe controller does not natively 4493support SGLs. 4494 4495The `identify` and `perf` NVMe examples were modified to add a consistent format for 4496specifying remote NVMe over Fabrics devices via the `-r` option. 4497This is implemented using the new `spdk_nvme_transport_id_parse()` function. 4498 4499### iSCSI Target 4500 4501The [Nvme] section of the configuration file was modified to remove the `BDF` directive 4502and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 4503devices can now be specified as the backing block device. A script to generate an 4504entire [Nvme] section based on the local NVMe devices attached was added at 4505`scripts/gen_nvme.sh`. 4506 4507### NVMe-oF Target 4508 4509The [Nvme] section of the configuration file was modified to remove the `BDF` directive 4510and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 4511devices can now be specified as the backing block device. A script to generate an 4512entire [Nvme] section based on the local NVMe devices attached was added at 4513`scripts/gen_nvme.sh`. 4514 4515## v16.12: NVMe over Fabrics host, hotplug, and multi-process 4516 4517### NVMe library 4518 4519The NVMe library has been changed to create its own request memory pool rather than 4520requiring the user to initialize the global `request_mempool` variable. Apps can be 4521updated by simply removing the initialization of `request_mempool`. Since the NVMe 4522library user no longer needs to know the size of the internal NVMe request 4523structure to create the pool, the `spdk_nvme_request_size()` function was also removed. 4524 4525The `spdk_nvme_ns_cmd_deallocate()` function was renamed and extended to become 4526`spdk_nvme_ns_cmd_dataset_management()`, which allows access to all of the NVMe 4527Dataset Management command's parameters. Existing callers can be updated to use 4528`spdk_nvme_ns_cmd_dataset_management()` with `SPDK_NVME_DSM_ATTR_DEALLOCATE` as the 4529`type` parameter. 4530 4531The NVMe library SGL callback prototype has been changed to return virtual addresses 4532rather than physical addresses. Callers of `spdk_nvme_ns_cmd_readv()` and 4533`spdk_nvme_ns_cmd_writev()` must update their `next_sge_fn` callbacks to match. 4534 4535The NVMe library now supports NVMe over Fabrics devices in addition to the existing 4536support for local PCIe-attached NVMe devices. For an example of how to enable 4537NVMe over Fabrics support in an application, see `examples/nvme/identify` and 4538`examples/nvme/perf`. 4539 4540Hot insert/remove support for NVMe devices has been added. To enable NVMe hotplug 4541support, an application should call the `spdk_nvme_probe()` function on a regular 4542basis to probe for new devices (reported via the existing `probe_cb` callback) and 4543removed devices (reported via a new `remove_cb` callback). Hotplug is currently 4544only supported on Linux with the `uio_pci_generic` driver, and newly-added NVMe 4545devices must be bound to `uio_pci_generic` by an external script or tool. 4546 4547Multiple processes may now coordinate and use a single NVMe device simultaneously 4548using [DPDK Multi-process Support](http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html). 4549 4550### NVMe over Fabrics target (`nvmf_tgt`) 4551 4552The `nvmf_tgt` configuration file format has been updated significantly to enable 4553new features. See the example configuration file `etc/spdk/nvmf.conf.in` for 4554more details on the new and changed options. 4555 4556The NVMe over Fabrics target now supports virtual mode subsystems, which allow the 4557user to export devices from the SPDK block device abstraction layer as NVMe over 4558Fabrics subsystems. Direct mode (raw NVMe device access) is also still supported, 4559and a single `nvmf_tgt` may export both types of subsystems simultaneously. 4560 4561### Block device abstraction layer (bdev) 4562 4563The bdev layer now supports scatter/gather read and write I/O APIs, and the NVMe 4564blockdev driver has been updated to support scatter/gather. Apps can use the 4565new scatter/gather support via the `spdk_bdev_readv()` and `spdk_bdev_writev()` 4566functions. 4567 4568The bdev status returned from each I/O has been extended to pass through NVMe 4569or SCSI status codes directly in cases where the underlying device can provide 4570a more specific status code. 4571 4572A Ceph RBD (RADOS Block Device) blockdev driver has been added. This allows the 4573`iscsi_tgt` and `nvmf_tgt` apps to export Ceph RBD volumes as iSCSI LUNs or 4574NVMe namespaces. 4575 4576### General changes 4577 4578`libpciaccess` has been removed as a dependency and DPDK PCI enumeration is 4579used instead. Prior to DPDK 16.07 enumeration by class code was not supported, 4580so for earlier DPDK versions, only Intel SSD DC P3x00 devices will be discovered 4581by the NVMe library. 4582 4583The `env` environment abstraction library has been introduced, and a default 4584DPDK-based implementation is provided as part of SPDK. The goal of the `env` 4585layer is to enable use of alternate user-mode memory allocation and PCI access 4586libraries. See `doc/porting.md` for more details. 4587 4588The build process has been modified to produce all of the library files in the 4589`build/lib` directory. This is intended to simplify the use of SPDK from external 4590projects, which can now link to SPDK libraries by adding the `build/lib` directory 4591to the library path via `-L` and linking the SPDK libraries by name (for example, 4592`-lspdk_nvme -lspdk_log -lspdk_util`). 4593 4594`nvmf_tgt` and `iscsi_tgt` now have a JSON-RPC interface, which allows the user 4595to query and modify the configuration at runtime. The RPC service is disabled by 4596default, since it currently does not provide any authentication or security 4597mechanisms; it should only be enabled on systems with controlled user access 4598behind a firewall. An example RPC client implemented in Python is provided in 4599`scripts/rpc.py`. 4600 4601## v16.08: iSCSI target, NVMe over Fabrics maturity 4602 4603This release adds a userspace iSCSI target. The iSCSI target is capable of exporting 4604NVMe devices over a network using the iSCSI protocol. The application is located 4605in app/iscsi_tgt and a documented configuration file can be found at etc/spdk/spdk.conf.in. 4606 4607This release also significantly improves the existing NVMe over Fabrics target. 4608 4609- The configuration file format was changed, which will require updates to 4610 any existing nvmf.conf files (see `etc/spdk/nvmf.conf.in`): 4611 - `SubsystemGroup` was renamed to `Subsystem`. 4612 - `AuthFile` was removed (it was unimplemented). 4613 - `nvmf_tgt` was updated to correctly recognize NQN (NVMe Qualified Names) 4614 when naming subsystems. The default node name was changed to reflect this; 4615 it is now "nqn.2016-06.io.spdk". 4616 - `Port` and `Host` sections were merged into the `Subsystem` section 4617 - Global options to control max queue depth, number of queues, max I/O 4618 size, and max in-capsule data size were added. 4619 - The Nvme section was removed. Now a list of devices is specified by 4620 bus/device/function directly in the Subsystem section. 4621 - Subsystems now have a Mode, which can be Direct or Virtual. This is an attempt 4622 to future-proof the interface, so the only mode supported by this release 4623 is "Direct". 4624- Many bug fixes and cleanups were applied to the `nvmf_tgt` app and library. 4625- The target now supports discovery. 4626 4627This release also adds one new feature and provides some better examples and tools 4628for the NVMe driver. 4629 4630- The Weighted Round Robin arbitration method is now supported. This allows 4631 the user to specify different priorities on a per-I/O-queue basis. To 4632 enable WRR, set the `arb_mechanism` field during `spdk_nvme_probe()`. 4633- A simplified "Hello World" example was added to show the proper way to use 4634 the NVMe library API; see `examples/nvme/hello_world/hello_world.c`. 4635- A test for measuring software overhead was added. See `test/lib/nvme/overhead`. 4636 4637## v16.06: NVMf userspace target 4638 4639This release adds a userspace NVMf (NVMe over Fabrics) target, conforming to the 4640newly-released NVMf 1.0/NVMe 1.2.1 specification. The NVMf target exports NVMe 4641devices from a host machine over the network via RDMA. Currently, the target is 4642limited to directly exporting physical NVMe devices, and the discovery subsystem 4643is not supported. 4644 4645This release includes a general API cleanup, including renaming all declarations 4646in public headers to include a `spdk` prefix to prevent namespace clashes with 4647user code. 4648 4649- NVMe 4650 - The `nvme_attach()` API was reworked into a new probe/attach model, which 4651 moves device detection into the NVMe library. The new API also allows 4652 parallel initialization of NVMe controllers, providing a major reduction in 4653 startup time when using multiple controllers. 4654 - I/O queue allocation was changed to be explicit in the API. Each function 4655 that generates I/O requests now takes a queue pair (`spdk_nvme_qpair *`) 4656 argument, and I/O queues may be allocated using 4657 `spdk_nvme_ctrlr_alloc_io_qpair()`. This allows more flexible assignment of 4658 queue pairs than the previous model, which only allowed a single queue 4659 per thread and limited the total number of I/O queues to the lowest number 4660 supported on any attached controller. 4661 - Added support for the Write Zeroes command. 4662 - `examples/nvme/perf` can now report I/O command latency from the 4663 the controller's viewpoint using the Intel vendor-specific read/write latency 4664 log page. 4665 - Added namespace reservation command support, which can be used to coordinate 4666 sharing of a namespace between multiple hosts. 4667 - Added hardware SGL support, which enables use of scattered buffers that 4668 don't conform to the PRP list alignment and length requirements on supported 4669 NVMe controllers. 4670 - Added end-to-end data protection support, including the ability to write and 4671 read metadata in extended LBA (metadata appended to each block of data in the 4672 buffer) and separate metadata buffer modes. 4673 See `spdk_nvme_ns_cmd_write_with_md()` and `spdk_nvme_ns_cmd_read_with_md()` 4674 for details. 4675- IOAT 4676 - The DMA block fill feature is now exposed via the `ioat_submit_fill()` 4677 function. This is functionally similar to `memset()`, except the memory is 4678 filled with an 8-byte repeating pattern instead of a single byte like memset. 4679- PCI 4680 - Added support for using DPDK for PCI device mapping in addition to the 4681 existing libpciaccess option. Using the DPDK PCI support also allows use of 4682 the Linux VFIO driver model, which means that SPDK userspace drivers will work 4683 with the IOMMU enabled. Additionally, SPDK applications may be run as an 4684 unprivileged user with access restricted to a specific set of PCIe devices. 4685 - The PCI library API was made more generic to abstract away differences 4686 between the underlying PCI access implementations. 4687 4688## v1.2.0: IOAT user-space driver 4689 4690This release adds a user-space driver with support for the Intel I/O Acceleration Technology 4691(I/OAT, also known as "Crystal Beach") DMA offload engine. 4692 4693- IOAT 4694 - New user-space driver supporting DMA memory copy offload 4695 - Example programs `ioat/perf` and `ioat/verify` 4696 - Kernel-mode DMA engine test driver `kperf` for performance comparison 4697- NVMe 4698 - Per-I/O flags for Force Unit Access (FUA) and Limited Retry 4699 - Public API for retrieving log pages 4700 - Reservation register/acquire/release/report command support 4701 - Scattered payload support - an alternate API to provide I/O buffers via a sequence of callbacks 4702 - Declarations and `nvme/identify` support for Intel SSD DC P3700 series vendor-specific log pages and features 4703- Updated to support DPDK 2.2.0 4704 4705## v1.0.0: NVMe user-space driver 4706 4707This is the initial open source release of the Storage Performance Development Kit (SPDK). 4708 4709Features: 4710 4711- NVMe user-space driver 4712- NVMe example programs 4713 - `examples/nvme/perf` tests performance (IOPS) using the NVMe user-space driver 4714 - `examples/nvme/identify` displays NVMe controller information in a human-readable format 4715- Linux and FreeBSD support 4716