1# BlobFS (Blobstore Filesystem) {#blobfs} 2 3# BlobFS Getting Started Guide {#blobfs_getting_started} 4 5# RocksDB Integration {#blobfs_rocksdb} 6 7Clone and build the SPDK repository as per https://github.com/spdk/spdk 8 9~~~{.sh} 10git clone https://github.com/spdk/spdk.git 11cd spdk 12./configure 13make 14~~~ 15 16Clone the RocksDB repository from the SPDK GitHub fork into a separate directory. 17Make sure you check out the `spdk-v5.6.1` branch. 18 19~~~{.sh} 20cd .. 21git clone -b spdk-v5.6.1 https://github.com/spdk/rocksdb.git 22~~~ 23 24Build RocksDB. Only the `db_bench` benchmarking tool is integrated with BlobFS. 25(Note: add `DEBUG_LEVEL=0` for a release build.) 26 27~~~{.sh} 28cd rocksdb 29make db_bench SPDK_DIR=path/to/spdk 30~~~ 31 32Copy `etc/spdk/rocksdb.conf.in` from the SPDK repository to `/usr/local/etc/spdk/rocksdb.conf`. 33 34~~~{.sh} 35cd ../spdk 36cp etc/spdk/rocksdb.conf.in /usr/local/etc/spdk/rocksdb.conf 37~~~ 38 39Append an NVMe section to the configuration file using SPDK's `gen_nvme.sh` script. 40 41~~~{.sh} 42scripts/gen_nvme.sh >> /usr/local/etc/spdk/rocksdb.conf 43~~~ 44 45Verify the configuration file has specified the correct NVMe SSD. 46If there are any NVMe SSDs you do not wish to use for RocksDB/SPDK testing, remove them from the configuration file. 47 48Make sure you have at least 5GB of memory allocated for huge pages. 49By default, the SPDK `setup.sh` script only allocates 2GB (1024 huge pages). 50The following will allocate 5GB worth of 2MB huge pages (in addition to binding the NVMe devices to uio/vfio). 51If using 1GB huge pages, adjust the `NRHUGE` value accordingly. 52 53~~~{.sh} 54NRHUGE=2560 scripts/setup.sh 55~~~ 56 57Create an empty SPDK blobfs for testing. 58 59~~~{.sh} 60test/lib/blobfs/mkfs/mkfs /usr/local/etc/spdk/rocksdb.conf Nvme0n1 61~~~ 62 63At this point, RocksDB is ready for testing with SPDK. Three `db_bench` parameters are used to configure SPDK: 64 651. `spdk` - Defines the name of the SPDK configuration file. If omitted, RocksDB will use the default PosixEnv implementation 66 instead of SpdkEnv. (Required) 672. `spdk_bdev` - Defines the name of the SPDK block device which contains the BlobFS to be used for testing. (Required) 683. `spdk_cache_size` - Defines the amount of userspace cache memory used by SPDK. Specified in terms of megabytes (MB). 69 Default is 4096 (4GB). (Optional) 70 71SPDK has a set of scripts which will run `db_bench` against a variety of workloads and capture performance and profiling 72data. The primary script is `test/blobfs/rocksdb/run_tests.sh`. 73 74# FUSE 75 76BlobFS provides a FUSE plug-in to mount an SPDK BlobFS as a kernel filesystem for inspection or debug purposes. 77The FUSE plug-in requires fuse3 and will be built automatically when fuse3 is detected on the system. 78 79~~~{.sh} 80test/lib/blobfs/fuse/fuse /usr/local/etc/spdk/rocksdb.conf Nvme0n1 /mnt/fuse 81~~~ 82 83Note that the FUSE plug-in has some limitations - see the list below. 84 85# Limitations 86 87* BlobFS has primarily been tested with RocksDB so far, so any use cases different from how RocksDB uses a filesystem 88 may run into issues. BlobFS will be tested in a broader range of use cases after this initial release. 89* Only a synchronous API is currently supported. An asynchronous API has been developed but not thoroughly tested 90 yet so is not part of the public interface yet. This will be added in a future release. 91* File renames are not atomic. This will be fixed in a future release. 92* BlobFS currently supports only a flat namespace for files with no directory support. Filenames are currently stored 93 as xattrs in each blob. This means that filename lookup is an O(n) operation. An SPDK btree implementation is 94 underway which will be the underpinning for BlobFS directory support in a future release. 95* Writes to a file must always append to the end of the file. Support for writes to any location within the file 96 will be added in a future release. 97