AOCL-Compression

AOCL-Compression is a software framework of various lossless compression and decompression methods tuned and optimized for AMD Zen based CPUs. This framework offers a single set of unified APIs for all the supported compression and decompression methods which facilitate the applications to easily integrate and use them. AOCL-Compression supports lz4, zlib/deflate, lzma, zstd, bzip2, snappy, and lz4hc based compression and decompression methods along with their native APIs. The library offers openMP based multi-threaded implementation of lz4, zlib, zstd and snappy compression methods. It supports the dynamic dispatcher feature that executes the most optimal function variant implemented using Function Multi-versioning thereby offering a single optimized library portable across different x86 CPU architectures. AOCL-Compression framework is developed in C for UNIX® and Windows® based systems. A test suite is provided for the validation and performance benchmarking of the supported compression and decompression methods. This suite also supports the benchmarking of IPP compression methods, such as, lz4, lz4hc, zlib and bzip2. The library build framework offers CTest based testing of the test cases implemented using GTest and the library test suite.

Installation

1. Download the latest stable release from the Github repository:
   https://github.com/amd/aocl-compression
2. Install CMake on the machine where the sources are to be compiled.
3. Make any one of the compilers GCC or Clang available on the machine.
4. Then, use the cmake based build system to compile and generate AOCL-Compression
   library and testsuite binary as explained below for Linux® and Windows® platforms.

Building on Linux

1. To create a build directory and configure the build system in it, run the following: cmake -B <build directory> <CMakeList.txt filepath> Additional options that can be specified for build configuration are: cmake -B <build directory> <CMakeList.txt filepath> -DCMAKE_INSTALL_PREFIX=<install path> -DCMAKE_BUILD_TYPE=<Debug or Release> -DBUILD_STATIC_LIBS=ON <Additional Library Build Options>

To use clang compiler for the build, specify -DCMAKE_C_COMPILER=clang as the option. 2. Compile using the following command: cmake --build <build directory> --target install -j The library is generated in "lib" directory.
The test bench executable is generated in "build".
The additional option --target install will install the library, and
interface header files in the installation path as specified with
-DCMAKE_INSTALL_PREFIX option or in the local system path.
The option -j will run the compilation process using multiple cores. 3. To uninstall the installed files, run the following custom command: cmake --build <build directory> --target uninstall To uninstall and then install the build package, run the following command: cmake --build <build directory> --target uninstall --target install -j -v The option -v will print verbose build logs on the console. 4. To clear or delete the build folder or files, manually remove the build directory or its files.

Building on Windows

As a prerequisite, make Microsoft Visual Studio® available along with
Desktop development with C++ toolset that includes the Clang compiler.

Building with Visual Studio IDE (GUI)

1. Launch CMake GUI and set the locations for source package and build output.
2. Click Configure option and select:
   • Generator as the Installed Microsoft Visual Studio Version
   • Platform as x64
   • Optional toolset as ClangCl
3. Select additional library config and build options.
4. Configure CMAKEINSTALLPREFIX appropriately.
5. Click Generate. Microsoft Visual Studio project is generated.
6. Click Open Project. Microsoft Visual Studio project for the source package is launched.
7. Build the entire solution or the required projects.

Building with Visual Studio IDE (command line)

1. Go to AOCL-Compression source package and create a folder named build.
2. Go to the build folder.
3. Use the following command to configure and build the library and test bench executable. cmake .. -T ClangCl -G <installed Visual Studio version> && cmake --build . --config Release --target INSTALL You can pass additional library configuration and build options in the command.

Additional Library Build Options

Use the following additional options to configure your build:

Option | Description ------------------------------------|---------------------------------------------------------------------------------------- AOCLLZ4OPTPREFETCHBACKWARDS | Enable LZ4 optimizations related to backward prefetching of data (Disabled by default) SNAPPYMATCHSKIPOPT | Enable Snappy match skipping optimization (Enabled by default) SNAPPYHIGHCOMPRESSION | Enable Snappy high compression to get better ratio by compromising on speed (Disabled by default) SNAPPYENABLEDECOMPRESSBRANCHLESS | Enable Snappy branchless decompression optimization (Disabled by default for GCC and enabled for all other compilers) LZ4FRAMEFORMATSUPPORT | Enable building LZ4 with Frame format and API support (Enabled by default) AOCLLZ4HCDISABLEPATTERNANALYSIS | Disable Pattern Analysis in LZ4HC for level 9 (Enabled by default) AOCLZSTDSEARCHSKIPOPT | Enable ZSTD match skipping optimization that steps more aggresively when matches are not found (Enabled by default) AOCLDECOMPRESSFAST | Enable fast decompression modes that might compromise on compression speed / ratio to produce streams that decompress faster. Supported values: {1,2,3} ZSTD, {1,2} Snappy, {1} LZ4. (Disabled by default) AOCLTESTCOVERAGE | Enable GTest, AOCL test bench and third party test bench based CTest suite (Disabled by default) AOCLENABLELOGFEATURE | Enables logging through environment variable AOCL_ENABLE_LOG (Disabled by default) CODECOVERAGE | Enable source code coverage. Only supported on Linux with the GCC compiler (Disabled by default) ASAN | Enable Address Sanitizer checks. Only supported on Linux/Debug build (Disabled by default) VALGRIND | Enable Valgrind checks. Only supported on Linux/Debug and incompatible with ASAN=ON (Disabled by default) BUILDDOC | Build documentation for this library (Disabled by default) BUILDEXAMPLE | Build examples for aocl-compression (Disabled by default) AOCLLZ4MATCHSKIPOPTLDSSTRAT1 | Enable LZ4 match skipping optimization strategy-1 based on a larger base step size applied for long distance search (Disabled by default) AOCLLZ4MATCHSKIPOPTLDSSTRAT2 | Enable LZ4 match skipping optimization strategy-2 by aggressively setting search distance on top of strategy-1. Preferred to be used with Silesia corpus (Disabled by default) AOCLLZ4NEWPRIMENUMBER | Enable the usage of a new prime number for LZ4 hashing function. Preferred to be used with Silesia corpus (Disabled by default) AOCLLZ4EXTRAHASHTABLEUPDATES | Enable storing of additional potential matches to improve compression ratio. Recommended for higher compressibility use cases (Disabled by default) AOCLLZ4HASHBITSUSED | Control the number of bits used for LZ4 hashing, allowed values are OFF, LOW (low perf gain and less CR regression) and HIGH (high perf gain and high CR regression) (LOW by default) AOCLEXCLUDEBZIP2 | Exclude BZIP2 compression method from the library build (Disabled by default) AOCLEXCLUDELZ4 | Exclude LZ4 compression method from the library build. LZ4HC also gets excluded (Disabled by default) AOCLEXCLUDELZ4HC | Exclude LZ4HC compression method from the library build (Disabled by default) AOCLEXCLUDELZMA | Exclude LZMA compression method from the library build (Disabled by default) AOCLEXCLUDESNAPPY | Exclude SNAPPY compression method from the library build (Disabled by default) AOCLEXCLUDEZLIB | Exclude ZLIB compression method from the library build (Disabled by default) AOCLEXCLUDEZSTD | Exclude ZSTD compression method from the library build (Disabled by default) AOCLXZUTILSLZMAAPIEXPERIMENTAL | Build with xz utils lzma APIs. Experimental feature with limited API support (Disabled by default) AOCLENABLETHREADS | Enable multi-threaded compression and decompression using SMP based openMP threads (Disabled by default) TESTCOVERAGETHIRDPARTY | Enable third party test bench based CTest suite (Disabled by default) NATIVEENABLETHREADS | Enable native multi-threaded compression for supported methods (Disabled by default) AOCLTESTFUZZER | Enable fuzz test along with GTest. Only supported on Linux with the Clang compiler (Disabled by default) AOCLTESTFUZZERWITHCORPUS | Run fuzz tests with corpus. Only supported on Linux with the Clang compiler (Disabled by default) ENABLEFASTMATH | Enable fast-math optimizations (Disabled by default) BUILD_UTILITY | Enable third party utility build: zstd (Disabled by default)" OFF)

• NOTE:
  
  1. ZLIB supports quicker compression strategy for Level 1 by trading off compression ratio. Enable it by
     setting environment variable AOCLZLIBQUICK_MODE=ON. It also improves performance for levels 2, 3 and 5
     while trading off compression ratio.
     

Running AOCL-Compression Test Bench On Linux

• CAUTION:
  Before running the test bench, check whether it points to the right library dependency.
  

Test bench supports several options to validate, benchmark or debug the supported compression methods. It uses the unified API set to invoke the compression methods supported by AOCL-Compression. Test bench can invoke and benchmark some of the IPP's compression methods as well.

• To check various options supported by the test bench, use one of the following commands:
  aocl_compression_bench -h
aocl_compression_bench --help

• To check all the supported compression methods, use the command:
  aocl_compression_bench -l

• To run the test bench with requested number of iterations, use the command:
  aocl_compression_bench -i

• To run the test bench to check the performance of all the supported compression
  and decompression methods for a given input file, use the command:
  aocl_compression_bench -a -p <input filename>

• To run the test bench to validate the outputs from all the supported compression
  and decompression methods for a given input file, use the command:
  aocl_compression_bench -a -t <input filename>

• To run the test bench to check the performance of a compression and decompression
  method for a given input file, use the command:
  aocl_compression_bench -ezstd:5:0 -p <input filename>
  Here, 5 is the level and 0 is the additional parameter passed to ZSTD method.

• To run the test bench to validate the output of a compression and decompression
  method for a given input file, use the command:
  aocl_compression_bench -ezstd:5:0 -t <input filename>
  Here, 5 is the level and 0 is the additional parameter passed to ZSTD method.

• To run the test bench with error/debug/trace/info logs, build the library by using -DAOCL_ENABLE_LOG_FEATURE=ON & set the environment variable AOCL_ENABLE_LOG to any of the following:
  

  • AOCL_ENABLE_LOG=ERR for Error logs.
  • AOCL_ENABLE_LOG=INFO for Error, Info logs.
  • AOCL_ENABLE_LOG=DEBUG for Error, Info, Debug logs.
  • AOCL_ENABLE_LOG=TRACE for Error, Info, Debug, Trace logs.
    Note: When building the library for highest performance, do not enable AOCL_ENABLE_LOG_FEATURE.

• To run the test bench but only compression or decompression
  for a given input file, use the command:
  aocl_compression_bench -rcompress <input filename> or
  aocl_compression_bench -rdecompress -ezstd <compressed input filename> or
  aocl_compression_bench -rdecompress -ezstd -t -f<uncompressed file for validation> <compressed input filename>
  Note: In -rdecompress mode, compression method must be specified using -e option.
  If validation of decompressed data is needed, specify -t and -f options additionally.

• To run the test bench and dump output data generated
  for a given input file, use the command:
  aocl_compression_bench -d<dump filename> -ezstd:1 <input filename> or
  aocl_compression_bench -d<dump filename> -rcompress -ezstd:1 <input filename> or
  aocl_compression_bench -d<dump filename> -rdecompress -ezstd <compressed input filename>
  Here, when -rcompress operation is selected, compressed file gets dumped
  and when -rdecompress operation is selected, decompressed file gets dumped.
  Method name and level must be specified using -e for default and -rcompress modes.
  Method name must be specified using -e for -rdecompress mode.
  

• To run the test bench and test native APIs, use the command:
  aocl_compression_bench -n -p <input filename>
  Other options -e, -i, -t, -r are supported when running with -n
  

• To run the test bench and test multi-threaded native APIs,
  for supported methods, use the command:
  aocl_compression_bench -e<method>:<level>:<num-of-workers> -n -p <input filename>
  

• To run the test bench and test native APIs with an external dictionary file
  for supported methods, use the command:
  aocl_compression_bench -e<method> -n -p -y<dictionary filename> <input filename>
  

• NOTE:
  

  1. Compression and decompression of large files (>1GB) are supported in the test bench.
     
  2. Decompression of compressed files (> 1GB) that are not generated by aocl-compression
     is not guaranteed by the test bench.
     

---

To test and benchmark the performance of IPP's compression methods, use the test bench option -c<path to IPP library method> along with other relevant options (as explained above). IPP's lz4, lz4hc, zlib and bzip2 methods are supported by the test bench. Check the following details for the exact steps: 1. Set the library path environment variable (export LDLIBRARYPATH on
Linux) to point to the installed IPP library path.
Alternatively, you can also run vars.sh that comes along with the
IPP installation to setup the environment variable. 2. Download lz4-1.9.3, zlib-1.2.11 and bzip2-1.0.8 source packages. 3. Apply IPP patch files using the command:
patch -p1 < path to corresponding patch file>

1. Build the patched IPP lz4, zlib and bzip2 libraries per the steps
   in the IPP readme files in the corresponding patch file
   locations for these compression methods.
2. Append the library path to -c option and pass it to executable as command line argument
   (Linux is only supported) for running patched IPP lz4, zlib and bzip2 libraries.
3. Run the test bench to benchmark the IPP library methods as follows: aocl_compression_bench -a -p -c/path/to/ipp_patch <input filename> aocl_compression_bench -elz4 -p -c/path/to/ipp_patch <input filename> aocl_compression_bench -elz4hc -p -c/path/to/ipp_patch <input filename> aocl_compression_bench -ezlib -p -c/path/to/ipp_patch <input filename> aocl_compression_bench -ebzip2 -p -c/path/to/ipp_patch <input filename>

Running AOCL-Compression Test Bench On Windows

• CAUTION:
  Before running the test bench, ensure it points to the right library dependencies for aocl_compression, openMP, etc.
  

Test bench on Windows supports all the user options as Linux, except for the -c option to link and test IPP compression methods. For more information on various user options, refer to the previous section on Linux. To set and launch the test bench with a specific user option, go to project aoclcompressionbench -> Properties -> Debugging; specify the user options and the input test file.

Running AOCL-Compression Examples

• CAUTION:
  Before running the example programs, ensure it points to the right library dependencies for aocl_compression, openMP, etc.
  

Example programs are provided for both unified API and native APIs of each compression method. The library should be built with -DBUILDEXAMPLE=ON. Other cmake options including -DAOCLENABLE_THREADS=ON can be enabled as desired.

• To run example program for unified API, use the command:
  example_unified_api <input filename>

• To run example program for LZ4 native API, use the command:
  example_LZ4_compress_default <input filename>

• To run example program that demonstrates obtaining format compliant compressed stream from multithreaded unified API, build the library by using -DAOCLENABLETHREADS=ON and run the command:
  example_aocl_llc_skip_rap_frame <input filename>

• To run example program that demonstrates obtaining format compliant gzip compressed stream from multithreaded API, build the library by using -DAOCLENABLETHREADS=ON and run the command:
  example_compress2_gzip <input filename>

Running tests with CTest

CTest is configured in CMake build system to run the test cases implemented with GTest and AOCL Test Bench for Silesia, Calgary, and Canterbury datasets. To enable testing with CTest, use AOCLTESTCOVERAGE option while configuring the CMake build.

Following are a few sample commands that can be executed in the build directory to run the test cases with CTest.

To run all the tests (GTest and Test bench)
ctest

To only run Test bench
ctest -R BENCH

To run GTest test cases for a specific method
ctest -R <METHOD_NAME_IN_CAPITALS>

Running fuzzer tests

To list all the fuzz tests available for a method, use the following command: <METHOD_GTEST_EXECUTABLE> --list_fuzz_tests example: zlib_gtest --list_fuzz_tests

Fuzzer test can be run in two modes:

1. Unit test mode: Default operation mode of AOCLTESTFUZZER. Can be run as part of ctest. No sanitizer and coverage instrumentation. ctest -R <TestSuiteName>.<FuzzTestName>
2. Fuzzing mode: Enabled with cmake option FUZZTESTFUZZINGMODE. Runs each fuzz test with sanitizer and coverage instrumentation

To run all fuzz tests for a specified duration, use the following command: <METHOD_GTEST_EXECUTABLE> --fuzz_for=<DURATION> example: zlib_gtest --fuzz_for=60s

To run a single fuzz test until a bug is found or until manually stopped: <METHOD_GTEST_EXECUTABLE> --fuzz=<TestSuiteName>.<FuzzTestName> example: zlib_gtest --fuzz=AOCL_Compression_zlib.compress2_fuzz

To run a single fuzz test by feeding in an external corpus of seeds: Enabled with cmake option AOCLTESTFUZZERWITHCORPUS. Place folders containing seed files in the directory pointed by environment variable AOCLFUZZCORPUSDIR. Sub-folders under this must be as follows: * /compressfuzz : Must contain uncompressed raw files for compress API fuzz tests. * /*fuzz : Folders with individual fuzz test names must contain compressed files for respective methods used for decompress API fuzz tests. Example: /LZ4decompresssafefuzz, /RawUncompressfuzz, etc Run the single fuzz test: `<METHODGTESTEXECUTABLE> --fuzz=. example:zlibgtest --fuzz=AOCLCompressionzlib.compress2fuzz` Additional seed properties can be specified by environment variables: * AOCLFUZZSIZEMAX : Max size in bytes to use for i/o buffers used in fuzz testing. * AOCLFUZZCPR_RATIO : Compression ratio estimate of compressed files used for decompress API fuzz tests.

Running source code coverage using GCOV

To measure source code coverage, use CODECOVERAGE option while configuring the CMake build. Run CMake with the custom target option 'code-coverage' to execute tests and generate code coverage data. The code coverage reports are generated in the build directory under subdirectory called 'coverage/htmlreport'. Open the HTML files in browser to view the coverage information.

Following is the sample command usage to run code coverage: cmake -B <build directory> <CMakeList.txt filepath> -DCMAKE_INSTALL_PREFIX=<install path> -DCMAKE_BUILD_TYPE=Debug -DBUILD_STATIC_LIBS=ON -DCODE_COVERAGE=ON <Additional Library Build Options> cmake --build <build directory> --target install code-coverage

Running Valgrind and ASAN memory checks using CTest

Use VALGRIND option for Valgrind memory check and ASAN option for ASAN memory check while configuring the CMake build. VALGRIND and ASAN options can not be enabled together.

Following are the commands to execute in the 'build' directory to run memory checks.

To run Valgrind memory check
ctest -T memcheck

To run ASAN memory check
ctest

Running Performance Benchmarking

Use test_speed.py script to benchmark performance and compare AOCL-Compression library with other compression libraries such as open-source reference or IPP. It generates summary reports describing compression/decompression speeds and compression ratio.

Following are a few sample commands to use the script available in the 'scripts' directory.

To print usage options
python3 test_speed.py --help

To run AOCL optimized vs Reference methods for lz4, snappy and zlib levels 1 and 2:
python3 test_speed.py --dataset $PATH_DATASETS_DIR -m lz4 snappy zlib:1 zlib:2 -cw vanilla

To run AOCL optimized vs IPP for lz4 method:
python3 test_speed.py --dataset $PATH_DATASETS_DIR -m lz4 -cw ipp --ipp $IPP_PATCHED_LZ4_LIBS_PATH

Generating Documentation

• To generate documentation, specify the -DBUILD_DOC=ON option while building.
• Documents will be generated in HTML format in the folder docs/html as doxygen output & docs/sphinx/html as sphinx output. Open the index.html file from respective folders in any browser to view the documentation.
• The following packages are expected before running CMake with -DBUILD_DOC=ON option:
  1. Doxygen.
  2. Python packages:
     • Sphinx
     • rocm_docs
     • breathe
     • myst_parser
• CMake halts if required packages are missing by providing directives for installing the absent packages.

Enabling/disabling optimizations

• AOCL optimizations can be disabled by setting the environment variable AOCLDISABLEOPT to ON.
• Reference code paths are taken in such a scenario.
• This needs to be set before launching the application for it to take effect.
• If optimization is turned off via aoclcompressiondesc::optOff (= 1) passed to aoclllcsetup(), then reference code paths are taken.
• If optimization is turned on via aoclcompressiondesc::optOff (= 0) passed to aoclllcsetup(), then AOCLDISABLEOPT is checked additionally to override aoclcompressiondesc::optOff value.

Enabling specific instructions (ISA)

• AOCL optimizations can be restricted to certain ISAs by setting the environment variable AOCLENABLEINSTRUCTIONS. Supported values are SSE2, AVX, AVX2 and AVX512.
• This ensures optimized code paths with ISAs above the set value are not taken. E.g. If it is set to AVX, no AVX2 and AVX512 optimized code paths are taken.
• This needs to be set before launching the application for it to take effect.
• It takes precedence over aoclcompressiondesc::optLevel setting passed to aoclllcsetup().
• Note: When calling aoclllcsetup() API from multiple threads, changing aoclcompressiondesc::optOff and aoclcompressiondesc::optLevel values between threads can lead to undefined behaviour.

Multi-threaded Compression and Decompression

• Parallel compression and decompression of lz4, lz4hc, zlib (zlib, deflate and gzip formats), zstd and snappy is implemented using openMP multi-threading. A RAP (random access point) frame is introduced in AOCL-Compression to support parallel decompression of the compressed streams/files. Use AOCLENABLETHREADS config option to enable the multi-threading.
• A stream compressed with multi-threaded AOCL-Compression library can be decompressed using any single-threaded standard decompressor by simply skipping the initial block of bytes containing the RAP frame present at the start of the stream.
• The multi-threaded compression support is optimally tuned for AMD CPUs on Linux® OS whereas this support is experimental for Windows® platforms.

CONTACTS

AOCL-Compression is developed and maintained by AMD.
For support, send an email to toolchainsupport@amd.com.