1# Peer-2-Peer DMAs {#peer_2_peer} 2 3Please note that the functionality discussed in this document is 4currently tagged as experimental. 5 6## In this document {#p2p_toc} 7 8* @ref p2p_overview 9* @ref p2p_nvme_api 10* @ref p2p_cmb_copy 11* @ref p2p_issues 12 13## Overview {#p2p_overview} 14 15Peer-2-Peer (P2P) is the concept of DMAing data directly from one PCI 16End Point (EP) to another without using a system memory buffer. The 17most obvious example of this from an SPDK perspective is using a NVMe 18Controller Memory Buffer (CMB) to enable direct copies of data between 19two NVMe SSDs. 20 21In this section of documentation we outline how to perform P2P 22operations in SPDK and outline some of the issues that can occur when 23performing P2P operations. 24 25## The P2P API for NVMe {#p2p_nvme_api} 26 27The functions that provide access to the NVMe CMBs for P2P 28capabilities are given in the table below. 29 30Key Functions | Description 31------------------------------------------- | ----------- 32spdk_nvme_ctrlr_map_cmb() | @copybrief spdk_nvme_ctrlr_map_cmb() 33spdk_nvme_ctrlr_unmap_cmb() | @copybrief spdk_nvme_ctrlr_unmap_cmb() 34spdk_nvme_ctrlr_get_regs_cmbsz() | @copybrief spdk_nvme_ctrlr_get_regs_cmbsz() 35 36## Determining device support {#p2p_support} 37 38SPDK's identify example application displays whether a device has a controller 39memory buffer and which operations it supports. Run it as follows: 40 41~~~{.sh} 42./build/examples/identify -r traddr:<pci id of ssd> 43~~~ 44 45## cmb_copy: An example P2P Application {#p2p_cmb_copy} 46 47Run the cmb_copy example application. 48 49~~~{.sh} 50./build/examples/cmb_copy -r <pci id of write ssd>-1-0-1 -w <pci id of write ssd>-1-0-1 -c <pci id of the ssd with cmb> 51~~~ 52This should copy a single LBA (LBA 0) from namespace 1 on the read 53NVMe SSD to LBA 0 on namespace 1 on the write SSD using the CMB as the 54DMA buffer. 55 56## Issues with P2P {#p2p_issues} 57 58* In some systems when performing peer-2-peer DMAs between PCIe EPs 59 that are directly connected to the Root Complex (RC) the DMA may 60 fail or the performance may not be great. Basically your milage may 61 vary. It is recommended that you use a PCIe switch (such as those 62 provided by Broadcom or Microsemi) as that is know to provide good 63 performance. 64* Even with a PCIe switch there may be occasions where peer-2-peer 65 DMAs fail to work. This is probably due to PCIe Access Control 66 Services (ACS) being enabled by the BIOS and/or OS. You can disable 67 ACS using setpci or via out of tree kernel patches that can be found 68 on the internet. 69* In more complex topologies involving several switches it may be 70 possible to construct multiple paths between EPs. This could lead to 71 TLP ordering problems. If you are working in these environments be 72 careful! 73