1b94155c8SStephen Bates# Peer-2-Peer DMAs {#peer_2_peer} 2b94155c8SStephen Bates 3b94155c8SStephen BatesPlease note that the functionality discussed in this document is 4b94155c8SStephen Batescurrently tagged as experimental. 5b94155c8SStephen Bates 61e1fd9acSwawryk## In this document {#p2p_toc} 7b94155c8SStephen Bates 8b94155c8SStephen Bates* @ref p2p_overview 9b94155c8SStephen Bates* @ref p2p_nvme_api 10b94155c8SStephen Bates* @ref p2p_cmb_copy 11b94155c8SStephen Bates* @ref p2p_issues 12b94155c8SStephen Bates 131e1fd9acSwawryk## Overview {#p2p_overview} 14b94155c8SStephen Bates 15b94155c8SStephen BatesPeer-2-Peer (P2P) is the concept of DMAing data directly from one PCI 16b94155c8SStephen BatesEnd Point (EP) to another without using a system memory buffer. The 17b94155c8SStephen Batesmost obvious example of this from an SPDK perspective is using a NVMe 18b94155c8SStephen BatesController Memory Buffer (CMB) to enable direct copies of data between 19b94155c8SStephen Batestwo NVMe SSDs. 20b94155c8SStephen Bates 21b94155c8SStephen BatesIn this section of documentation we outline how to perform P2P 22b94155c8SStephen Batesoperations in SPDK and outline some of the issues that can occur when 23b94155c8SStephen Batesperforming P2P operations. 24b94155c8SStephen Bates 251e1fd9acSwawryk## The P2P API for NVMe {#p2p_nvme_api} 26b94155c8SStephen Bates 27b94155c8SStephen BatesThe functions that provide access to the NVMe CMBs for P2P 28b94155c8SStephen Batescapabilities are given in the table below. 29b94155c8SStephen Bates 30b94155c8SStephen BatesKey Functions | Description 31b94155c8SStephen Bates------------------------------------------- | ----------- 32265a8436SBen Walkerspdk_nvme_ctrlr_map_cmb() | @copybrief spdk_nvme_ctrlr_map_cmb() 33265a8436SBen Walkerspdk_nvme_ctrlr_unmap_cmb() | @copybrief spdk_nvme_ctrlr_unmap_cmb() 341de99be6SBen Walkerspdk_nvme_ctrlr_get_regs_cmbsz() | @copybrief spdk_nvme_ctrlr_get_regs_cmbsz() 351de99be6SBen Walker 361e1fd9acSwawryk## Determining device support {#p2p_support} 371de99be6SBen Walker 381de99be6SBen WalkerSPDK's identify example application displays whether a device has a controller 391de99be6SBen Walkermemory buffer and which operations it supports. Run it as follows: 401de99be6SBen Walker 411de99be6SBen Walker~~~{.sh} 42017f65b5SBen Walkerspdk_nvme_identify -r traddr:<pci id of ssd> 431de99be6SBen Walker~~~ 44b94155c8SStephen Bates 451e1fd9acSwawryk## cmb_copy: An example P2P Application {#p2p_cmb_copy} 46b94155c8SStephen Bates 47b94155c8SStephen BatesRun the cmb_copy example application. 48b94155c8SStephen Bates 49b94155c8SStephen Bates~~~{.sh} 5085bdd43bSBen Walker./build/examples/cmb_copy -r <pci id of write ssd>-1-0-1 -w <pci id of write ssd>-1-0-1 -c <pci id of the ssd with cmb> 51b94155c8SStephen Bates~~~ 52b94155c8SStephen BatesThis should copy a single LBA (LBA 0) from namespace 1 on the read 53b94155c8SStephen BatesNVMe SSD to LBA 0 on namespace 1 on the write SSD using the CMB as the 54b94155c8SStephen BatesDMA buffer. 55b94155c8SStephen Bates 561e1fd9acSwawryk## Issues with P2P {#p2p_issues} 57b94155c8SStephen Bates 58b94155c8SStephen Bates* In some systems when performing peer-2-peer DMAs between PCIe EPs 59b94155c8SStephen Bates that are directly connected to the Root Complex (RC) the DMA may 60*34edd9f1SKamil Godzwon fail or the performance may not be great. Basically your mileage may 61b94155c8SStephen Bates vary. It is recommended that you use a PCIe switch (such as those 62b94155c8SStephen Bates provided by Broadcom or Microsemi) as that is know to provide good 63b94155c8SStephen Bates performance. 64b94155c8SStephen Bates* Even with a PCIe switch there may be occasions where peer-2-peer 651f813ec3SChen Wang DMAs fail to work. This is probably due to PCIe Access Control 66b94155c8SStephen Bates Services (ACS) being enabled by the BIOS and/or OS. You can disable 67b94155c8SStephen Bates ACS using setpci or via out of tree kernel patches that can be found 68b94155c8SStephen Bates on the internet. 69b94155c8SStephen Bates* In more complex topologies involving several switches it may be 70b94155c8SStephen Bates possible to construct multiple paths between EPs. This could lead to 71b94155c8SStephen Bates TLP ordering problems. If you are working in these environments be 72b94155c8SStephen Bates careful! 73