1*3117ece4Schristos# Parallel Zstandard (PZstandard) 2*3117ece4Schristos 3*3117ece4SchristosParallel Zstandard is a Pigz-like tool for Zstandard. 4*3117ece4SchristosIt provides Zstandard format compatible compression and decompression that is able to utilize multiple cores. 5*3117ece4SchristosIt breaks the input up into equal sized chunks and compresses each chunk independently into a Zstandard frame. 6*3117ece4SchristosIt then concatenates the frames together to produce the final compressed output. 7*3117ece4SchristosPzstandard will write a 12 byte header for each frame that is a skippable frame in the Zstandard format, which tells PZstandard the size of the next compressed frame. 8*3117ece4SchristosPZstandard supports parallel decompression of files compressed with PZstandard. 9*3117ece4SchristosWhen decompressing files compressed with Zstandard, PZstandard does IO in one thread, and decompression in another. 10*3117ece4Schristos 11*3117ece4Schristos## Usage 12*3117ece4Schristos 13*3117ece4SchristosPZstandard supports the same command line interface as Zstandard, but also provides the `-p` option to specify the number of threads. 14*3117ece4SchristosDictionary mode is not currently supported. 15*3117ece4Schristos 16*3117ece4SchristosBasic usage 17*3117ece4Schristos 18*3117ece4Schristos pzstd input-file -o output-file -p num-threads -# # Compression 19*3117ece4Schristos pzstd -d input-file -o output-file -p num-threads # Decompression 20*3117ece4Schristos 21*3117ece4SchristosPZstandard also supports piping and fifo pipes 22*3117ece4Schristos 23*3117ece4Schristos cat input-file | pzstd -p num-threads -# -c > /dev/null 24*3117ece4Schristos 25*3117ece4SchristosFor more options 26*3117ece4Schristos 27*3117ece4Schristos pzstd --help 28*3117ece4Schristos 29*3117ece4SchristosPZstandard tries to pick a smart default number of threads if not specified (displayed in `pzstd --help`). 30*3117ece4SchristosIf this number is not suitable, during compilation you can define `PZSTD_NUM_THREADS` to the number of threads you prefer. 31*3117ece4Schristos 32*3117ece4Schristos## Benchmarks 33*3117ece4Schristos 34*3117ece4SchristosAs a reference, PZstandard and Pigz were compared on an Intel Core i7 @ 3.1 GHz, each using 4 threads, with the [Silesia compression corpus](https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia). 35*3117ece4Schristos 36*3117ece4SchristosCompression Speed vs Ratio with 4 Threads | Decompression Speed with 4 Threads 37*3117ece4Schristos------------------------------------------|----------------------------------- 38*3117ece4Schristos |  39*3117ece4Schristos 40*3117ece4SchristosThe test procedure was to run each of the following commands 2 times for each compression level, and take the minimum time. 41*3117ece4Schristos 42*3117ece4Schristos time pzstd -# -p 4 -c silesia.tar > silesia.tar.zst 43*3117ece4Schristos time pzstd -d -p 4 -c silesia.tar.zst > /dev/null 44*3117ece4Schristos 45*3117ece4Schristos time pigz -# -p 4 -k -c silesia.tar > silesia.tar.gz 46*3117ece4Schristos time pigz -d -p 4 -k -c silesia.tar.gz > /dev/null 47*3117ece4Schristos 48*3117ece4SchristosPZstandard was tested using compression levels 1-19, and Pigz was tested using compression levels 1-9. 49*3117ece4SchristosPigz cannot do parallel decompression, it simply does each of reading, decompression, and writing on separate threads. 50*3117ece4Schristos 51*3117ece4Schristos## Tests 52*3117ece4Schristos 53*3117ece4SchristosTests require that you have [gtest](https://github.com/google/googletest) installed. 54*3117ece4SchristosSet `GTEST_INC` and `GTEST_LIB` in `Makefile` to specify the location of the gtest headers and libraries. 55*3117ece4SchristosAlternatively, run `make googletest`, which will clone googletest and build it. 56*3117ece4SchristosRun `make tests && make check` to run tests. 57
