Kernel Float

Kernel Float is a header-only library for CUDA/HIP that simplifies working with vector types and reduced precision floating-point arithmetic in GPU code.

Summary

CUDA/HIP natively offers several reduced precision floating-point types (__half, __nv_bfloat16, __nv_fp8_e4m3, __nv_fp8_e5m2) and vector types (e.g., __half2, __nv_fp8x4_e4m3, float3). However, working with these types is cumbersome: mathematical operations require intrinsics (e.g., __hadd2 performs addition for __half2), type conversion is awkward (e.g., __nv_cvt_halfraw2_to_fp8x2 converts float16 to float8), and some functionality is missing (e.g., one cannot convert a __half to __nv_bfloat16).

Kernel Float resolves this by offering a single data type kernel_float::vec<T, N> that stores N elements of type T. Internally, the data is stored as a fixed-sized array of elements. Operator overloading (like +, *, &&) has been implemented such that the most optimal intrinsic for the available types is selected automatically. Many mathematical functions (like log, exp, sin) and common operations (such as sum, range, for_each) are also available.

Using Kernel Float, developers avoid the complexity of reduced precision floating-point types in CUDA and can focus on their applications.

Features

In a nutshell, Kernel Float offers the following features:

• Single type vec<T, N> that unifies all vector types.
• Operator overloading to simplify programming.
• Support for half (16 bit) floating-point arithmetic, with a fallback to single precision for unsupported operations.
• Support for quarter (8 bit) floating-point types.
• Easy integration as a single header file.
• Written for C++17.
• Compatible with NVCC (NVIDIA Compiler) and NVRTC (NVIDIA Runtime Compilation).
• Compatible with HIPCC (AMD HIP Compiler)

Example

Check out the examples directory for some examples.

Below shows a simple example of a CUDA kernel that adds a constant to the input array and writes the results to the output array. Each thread processes two elements. Notice how easy it would be to change the precision (for example, double to half) or the vector size (for example, 4 instead of 2 items per thread).

```cpp

include "kernel_float.h"

namespace kf = kernel_float;

global void kernel(kf::vecptr input, int constant, kf::vecptr output) { int i = blockIdx.x * blockDim.x + threadIdx.x; output[i] += input[i] * constant; }

```

Here is how the same kernel would look for CUDA without Kernel Float.

```cpp global void kernel(const half* input, double constant, float* output) { int i = blockIdx.x * blockDim.x + threadIdx.x; _half in0 = input[2 * i + 0]; _half in1 = input[2 * i + 1]; _half2 a = _halves2half2(in0, in1); _half b = _int2halfrn(constant); _half2 c = _half2half2(b); _half2 d = _hmul2(a, c); _half e = _low2half(d); _half f = _high2half(d); float out0 = _half2float(e); float out1 = __half2float(f); output[2 * i + 0] += out0; output[2 * i + 1] += out1; }

```

Even though the second kernel looks a lot more complex, the PTX code generated by these two kernels is nearly identical.

Installation

This is a header-only library. Copy the file single_include/kernel_float.h to your project and include it:

```cpp

include "kernel_float.h"

```

Use the provided Makefile to generate this single-include header file if it is outdated:

make

Documentation

See the documentation for the API reference of all functionality.

License

Licensed under Apache 2.0. See LICENSE.

Related Work

• Kernel Tuner
• Kernel Launcher