FluidX3D

The fastest and most memory efficient lattice Boltzmann CFD software, running on all GPUs and CPUs via OpenCL. Free for non-commercial use.

(click on images to show videos on YouTube)

How to get started?

Read the FluidX3D Documentation!

Compute Features - Getting the Memory Problem under Control

• CFD model: lattice Boltzmann method (LBM)

  • streaming (part 2/2)

    f₀^temp(x,t) = f₀(x, t)
    f_i^temp(x,t) = f_{(t%2 ? i : (i%2 ? i+1 : i-1))}(i%2 ? x : x-e_i, t)   for   i ∈ [1, q-1]

  • collision

    ρ(x,t) = (Σ_i f_i^temp(x,t)) + 1
    
    u(x,t) = ¹∕_ρ(x,t) Σ_i c_i f_i^temp(x,t)
    
    f_i^eq-shifted(x,t) = w_i ρ · (^(u_°c_i)2∕_(2c⁴) - ^(u_°u)∕_(2c²) + ^(u_°c_i)∕_c²) + w_i (ρ-1)
    
    f_i^temp(x, t+Δt) = f_i^temp(x,t) + Ω_i(f_i^temp(x,t), f_i^eq-shifted(x,t), τ)

  • streaming (part 1/2)

    f₀(x, t+Δt) = f₀^temp(x, t+Δt)
    f_{(t%2 ? (i%2 ? i+1 : i-1) : i)}(i%2 ? x+e_i : x, t+Δt) = f_i^temp(x, t+Δt)   for   i ∈ [1, q-1]

  • variables and notation

  | variable | SI units | defining equation | description | | :------------------: | :---------------------------------: | :-------------------------------------------------: | :------------------------------------------------------------------------------ | | | | | | | x | m | x = (x,y,z)^T | 3D position in Cartesian coordinates | | t | s | - | time | | ρ | ^kg∕_m³ | ρ = (Σ_i f_i)+1 | mass density of fluid | | p | ^kg∕_m s² | p = c² ρ | pressure of fluid | | u | ^m∕_s | u = ¹∕_ρ Σ_i c_i f_i | velocity of fluid | | ν | ^m²∕_s | ν = ^μ∕_ρ | kinematic shear viscosity of fluid | | μ | ^kg∕_m s | μ = ρ ν | dynamic viscosity of fluid | | | | | | | f_i | ^kg∕_m³ | - | shifted density distribution functions (DDFs) | | Δx | m | Δx = 1 | lattice constant (in LBM units) | | Δt | s | Δt = 1 | simulation time step (in LBM units) | | c | ^m∕_s | c = ¹∕_√3 ^Δx∕_Δt | lattice speed of sound (in LBM units) | | i | 1 | 0 ≤ i < q | LBM streaming direction index | | q | 1 | q ∈ { 9,15,19,27 } | number of LBM streaming directions | | e_i | m | D2Q9 / D3Q15/19/27 | LBM streaming directions | | c_i | ^m∕_s | c_i = ^e_i∕_Δt | LBM streaming velocities | | w_i | 1 | Σ_i w_i = 1 | LBM velocity set weights | | Ω_i | ^kg∕_m³ | SRT or TRT | LBM collision operator | | τ | s | τ = ^ν∕_c² + ^Δt∕₂ | LBM relaxation time |

  - velocity sets: D2Q9, D3Q15, D3Q19 (default), D3Q27 - collision operators: single-relaxation-time (SRT/BGK) (default), two-relaxation-time (TRT) - DDF-shifting and other algebraic optimization to minimize round-off error

• optimized to minimize VRAM footprint to 1/6 of other LBM codes

  • traditional LBM (D3Q19) with FP64 requires ~344 Bytes/cell
    
  • 🟧🟧🟧🟧🟧🟧🟧🟧🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟨🟨🟨🟨🟨🟨🟨🟨🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥
    (density 🟧, velocity 🟦, flags 🟨, 2 copies of DDFs 🟩/🟥; each square = 1 Byte)
  • allows for 3 Million cells per 1 GB VRAM
  • FluidX3D (D3Q19) requires only 55 Bytes/cell with Esoteric-Pull+FP16
    
  • 🟧🟧🟧🟧🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟨🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩
    (density 🟧, velocity 🟦, flags 🟨, DDFs 🟩; each square = 1 Byte)
  • allows for 19 Million cells per 1 GB VRAM
  • in-place streaming with Esoteric-Pull: eliminates redundant copy of density distribution functions (DDFs) in memory; almost cuts memory demand in half and slightly increases performance due to implicit bounce-back boundaries; offers optimal memory access patterns for single-cell in-place streaming
  • decoupled arithmetic precision (FP32) and memory precision (FP32 or FP16S or FP16C): all arithmetic is done in FP32 for compatibility on all hardware, but DDFs in memory can be compressed to FP16S or FP16C: almost cuts memory demand in half again and almost doubles performance, without impacting overall accuracy for most setups
  • only 8 flag bits per lattice point (can be used independently / at the same time)

    • TYPE_S (stationary or moving) solid boundaries
    • TYPE_E equilibrium boundaries (inflow/outflow)
    • TYPE_T temperature boundaries
    • TYPE_F free surface (fluid)
    • TYPE_I free surface (interface)
    • TYPE_G free surface (gas)
    • TYPE_X remaining for custom use or further extensions
    • TYPE_Y remaining for custom use or further extensions

  - large cost saving: comparison of maximum single-GPU grid resolution for D3Q19 LBM

  | GPU VRAM capacity | 1 GB | 2 GB | 3 GB | 4 GB | 6 GB | 8 GB | 10 GB | 11 GB | 12 GB | 16 GB | 20 GB | 24 GB | 32 GB | 40 GB | 48 GB | 64 GB | 80 GB | 94 GB | 128 GB | 180 GB | 192 GB | 256 GB | 288 GB | 6 TB | | :------------------------------- | --------: | --------: | --------: | --------: | --------: | --------: | ---------: | ---------: | ---------: | ---------: | ---------: | ---------: | ---------: | ---------: | ---------: | ---------: | ---------: | ---------: | ----------: | ----------: | ----------: | ----------: | ----------: | --------: | | approximate GPU price | $25
  GT 210 | $25
  GTX 950 | $12
  GTX 1060 | $50
  GT 730 | $35
  GTX 1060 | $70
  RX 470 | $500
  RTX 3080 | $240
  GTX 1080 Ti | $75
  Tesla M40 | $75
  Instinct MI25 | $900
  RX 7900 XT | $205
  Tesla P40 | $600
  Instinct MI60 | $5500
  A100 | $2400
  RTX 8000 | $10k
  Instinct MI210 | $11k
  A100 | >$40k
  H100 NVL | ?
  GPU Max 1550 | ~$80k
  B200 | ~$15k
  MI300X | ~$25k
  MI325X | ~$25k
  MI350X | ~$80k
  2x Xeon 6960P | | traditional LBM (FP64) | 144³ | 182³ | 208³ | 230³ | 262³ | 288³ | 312³ | 322³ | 330³ | 364³ | 392³ | 418³ | 460³ | 494³ | 526³ | 578³ | 624³ | 658³ | 730³ | 818³ | 836³ | 920³ | 956³ | 2654³ | | FluidX3D (FP32/FP32) | 224³ | 282³ | 322³ | 354³ | 406³ | 448³ | 482³ | 498³ | 512³ | 564³ | 608³ | 646³ | 710³ | 766³ | 814³ | 896³ | 966³ | 1018³ | 1130³ | 1266³ | 1292³ | 1422³ | 1480³ | 4106³ | | FluidX3D (FP32/FP16) | 266³ | 336³ | 384³ | 424³ | 484³ | 534³ | 574³ | 594³ | 610³ | 672³ | 724³ | 770³ | 848³ | 912³ | 970³ | 1068³ | 1150³ | 1214³ | 1346³ | 1508³ | 1540³ | 1696³ | 1764³ | 4892³ |

-

Solving the Visualization Problem

• FluidX3D can do simulations so large that storing the volumetric data for later rendering becomes unmanageable (like 120GB for a single frame, hundreds of TeraByte for a video)
• instead, FluidX3D allows rendering raw simulation data directly in VRAM, so no large volumetric files have to be exported to the hard disk (see my technical talk)
• the rendering is so fast that it works interactively in real time for both rasterization and raytracing
• rasterization and raytracing are done in OpenCL and work on all GPUs, even the ones without RTX/DXR raytracing cores or without any rendering hardware at all (like A100, MI200, ...)
• if no monitor is available (like on a remote Linux server), there is an ASCII rendering mode to interactively visualize the simulation in the terminal (even in WSL and/or through SSH)
• rendering is fully multi-GPU-parallelized via seamless domain decomposition rasterization
• with interactive graphics mode disabled, image resolution can be as large as VRAM allows for (4K/8K/16K and above)
• (interacitive) visualization modes:
  • flag wireframe / solid surface (and force vectors on solid cells or surface pressure if the extension is used)
  • velocity field (with slice mode)
  • streamlines (with slice mode)
  • velocity-colored Q-criterion isosurface
  • rasterized free surface with marching-cubes
  • raytraced free surface with fast ray-grid traversal and marching-cubes, either 1-4 rays/pixel or 1-10 rays/pixel

Solving the Compatibility Problem

• FluidX3D is written in OpenCL 1.2, so it runs on all hardware from all vendors (Nvidia, AMD, Intel, ...):
  • world's fastest datacenter GPUs: B200, MI300X, H200, H100 (NVL), A100, MI200, MI100, V100(S), GPU Max 1100, ...
  • gaming GPUs (desktop/laptop): Nvidia GeForce, AMD Radeon, Intel Arc
  • professional/workstation GPUs: Nvidia Quadro, AMD Radeon Pro / FirePro, Intel Arc Pro
  • integrated GPUs
  • CPUs (requires installation of Intel CPU Runtime for OpenCL)
  • Intel Xeon Phi (requires installation of Intel CPU Runtime for OpenCL)
  • smartphone ARM GPUs
• native cross-vendor multi-GPU implementation
  • uses PCIe communication, so no SLI/Crossfire/NVLink/InfinityFabric required
  • single-node parallelization, so no MPI installation required
  • GPUs don't even have to be from the same vendor, but similar memory capacity and bandwidth are recommended
• works on Windows and Linux with C++17, with limited support also for macOS and Android
• supports importing and voxelizing triangle meshes from binary .stl files, with fast GPU voxelization
• supports exporting volumetric data as binary .vtk files
• supports exporting triangle meshes as binary .vtk files
• supports exporting rendered images as .png/.qoi/.bmp files; encoding runs in parallel on the CPU while the simulation on GPU can continue without delay

Single-GPU/CPU Benchmarks

Here are performance benchmarks on various hardware in MLUPs/s, or how many million lattice cells are updated per second. The settings used for the benchmark are D3Q19 SRT with no extensions enabled (only LBM with implicit mid-grid bounce-back boundaries) and the setup consists of an empty cubic box with sufficient size (typically 256³). Without extensions, a single lattice cell requires: - a memory capacity of 93 (FP32/FP32) or 55 (FP32/FP16) Bytes - a memory bandwidth of 153 (FP32/FP32) or 77 (FP32/FP16) Bytes per time step - 363 (FP32/FP32) or 406 (FP32/FP16S) or 1275 (FP32/FP16C) FLOPs per time step (FP32+INT32 operations counted combined)

In consequence, the arithmetic intensity of this implementation is 2.37 (FP32/FP32) or 5.27 (FP32/FP16S) or 16.56 (FP32/FP16C) FLOPs/Byte. So performance is only limited by memory bandwidth. The table in the left 3 columns shows the hardware specs as found in the data sheets (theoretical peak FP32 compute performance, memory capacity, theoretical peak memory bandwidth). The right 3 columns show the measured FluidX3D performance for FP32/FP32, FP32/FP16S, FP32/FP16C floating-point precision settings, with the (roofline model efficiency) in round brackets, indicating how much % of theoretical peak memory bandwidth are being used.

If your GPU/CPU is not on the list yet, you can report your benchmarks here.

```mermaid gantt

title FluidX3D Performance [MLUPs/s] - FP32 arithmetic, (fastest of FP32/FP16S/FP16C) memory storage dateFormat X axisFormat %s %%{ init: { "gantt": { 'titleTopMargin': 42, 'topPadding': 70, 'leftPadding': 260, 'rightPadding': 5, 'sectionFontSize': 20, 'fontSize': 20, 'barHeight': 20, 'barGap': 3, 'numberSectionStyles': 2 }, 'theme': 'forest', 'themeVariables': { 'sectionBkgColor': '#99999999', 'altSectionBkgColor': '#00000000', 'titleColor': '#AFAFAF', 'textColor': '#AFAFAF', 'taskTextColor': 'black', 'taskBorderColor': '#487E3A' } } }%%

section MI300X 41327 :crit, 0, 41327 section MI250 (1 GCD) 9030 :crit, 0, 9030 section MI210 9547 :crit, 0, 9547 section MI100 8542 :crit, 0, 8542 section MI60 5111 :crit, 0, 5111 section MI50 32GB 8477 :crit, 0, 8477 section Radeon VII 7778 :crit, 0, 7778 section GPU Max 1100 6303 :done, 0, 6303 section B200 SXM6 180GB 55609 : 0, 55609 section H200 SXM5 141GB 36610 : 0, 36610 section GH200 94GB GPU 34689 : 0, 34689 section H100 NVL 32922 : 0, 32922 section H100 SXM5 80GB HBM3 29561 : 0, 29561 section H100 PCIe 80GB HBM2e 20624 : 0, 20624 section A100 SXM4 80GB 18448 : 0, 18448 section A100 PCIe 80GB 17896 : 0, 17896 section PG506-242/243 15654 : 0, 15654 section A100 SXM4 40GB 16013 : 0, 16013 section A100 PCIe 40GB 16035 : 0, 16035 section CMP 170HX 12392 : 0, 12392 section A30 9721 : 0, 9721 section V100 SXM2 32GB 8947 : 0, 8947 section V100 PCIe 16GB 10325 : 0, 10325 section GV100 6641 : 0, 6641 section Titan V 7253 : 0, 7253 section P100 PCIe 16GB 5950 : 0, 5950 section P100 PCIe 12GB 4141 : 0, 4141 section GTX TITAN 2500 : 0, 2500 section K40m 1868 : 0, 1868 section K80 (1 GPU) 1642 : 0, 1642 section K20c 1507 : 0, 1507

section RX 9070 XT 6688 :crit, 0, 6688 section RX 9070 6019 :crit, 0, 6019 section RX 7900 XTX 7716 :crit, 0, 7716 section PRO W7900 5939 :crit, 0, 5939 section RX 7900 XT 5986 :crit, 0, 5986 section RX 7800 XT 3105 :crit, 0, 3105 section PRO W7800 4426 :crit, 0, 4426 section RX 7900 GRE 4570 :crit, 0, 4570 section PRO W7700 2943 :crit, 0, 2943 section RX 7700 XT 2828 :crit, 0, 2828 section RX 7600 2561 :crit, 0, 2561 section PRO W7600 2287 :crit, 0, 2287 section PRO W7500 1682 :crit, 0, 1682 section RX 6900 XT 4227 :crit, 0, 4227 section RX 6800 XT 4241 :crit, 0, 4241 section PRO W6800 3361 :crit, 0, 3361 section RX 6700 XT 2908 :crit, 0, 2908 section RX 6750 GRE 12GB 2848 :crit, 0, 2848 section RX 6800M 3213 :crit, 0, 3213 section RX 6700M 2429 :crit, 0, 2429 section RX 6600 1839 :crit, 0, 1839 section RX 6500 XT 1030 :crit, 0, 1030 section RX 5700 XT 3253 :crit, 0, 3253 section RX 5700 3167 :crit, 0, 3167 section RX 5600 XT 2214 :crit, 0, 2214 section RX Vega 64 3227 :crit, 0, 3227 section RX 590 1688 :crit, 0, 1688 section RX 580 4GB 1848 :crit, 0, 1848 section RX 580 2048SP 8GB 1622 :crit, 0, 1622 section RX 480 8GB 1908 :crit, 0, 1908 section Pro WX 5100 1604 :crit, 0, 1604 section R9 Fury X 2880 :crit, 0, 2880 section R9 Nano 2761 :crit, 0, 2761 section R9 390X 2217 :crit, 0, 2217 section R9 290X 1699 :crit, 0, 1699 section R9 290 1647 :crit, 0, 1647 section HD 7970 1563 :crit, 0, 1563 section HD 7870 868 :crit, 0, 868 section HD 7850 635 :crit, 0, 635 section HD 5870 257 :crit, 0, 257 section HD 5770 132 :crit, 0, 132 section FirePro V5800 108 :crit, 0, 108 section Arc B580 LE 4979 :done, 0, 4979 section Arc A770 LE 4568 :done, 0, 4568 section Arc A750 LE 4314 :done, 0, 4314 section Arc A580 3889 :done, 0, 3889 section Arc Pro A40 985 :done, 0, 985 section Arc A380 1115 :done, 0, 1115 section RTX PRO 6000 Blackwell 20841 : 0, 20841 section RTX 5090 19141 : 0, 19141 section RTX 5080 10304 : 0, 10304 section RTX 5070 7238 : 0, 7238 section RTX 4090 11496 : 0, 11496 section RTX 6000 Ada 10293 : 0, 10293 section L40S 7637 : 0, 7637 section L40 7945 : 0, 7945 section RTX 4080 Super 8218 : 0, 8218 section RTX 4080 7933 : 0, 7933 section RTX 4070 Ti Super 7295 : 0, 7295 section RTX 4090M 6901 : 0, 6901 section RTX 4070 Super 5554 : 0, 5554 section RTX 4070 5016 : 0, 5016 section RTX 4080M 5114 : 0, 5114 section RTX 4000 Ada 4221 : 0, 4221 section L4 2857 : 0, 2857 section RTX 4060 3124 : 0, 3124 section RTX 4070M 3092 : 0, 3092 section RTX 2000 Ada 2526 : 0, 2526 section RTX 3090 Ti 10956 : 0, 10956 section RTX 3090 10732 : 0, 10732 section RTX 3080 Ti 9832 : 0, 9832 section RTX 3080 12GB 9657 : 0, 9657 section RTX A6000 8814 : 0, 8814 section RTX A5000 8617 : 0, 8617 section RTX 3080 10GB 8118 : 0, 8118 section RTX A40 6622 : 0, 6622 section RTX 3070 Ti 6807 : 0, 6807 section RTX A10 5741 : 0, 5741 section RTX 3080M Ti 5908 : 0, 5908 section RTX 3070 5096 : 0, 5096 section RTX 3060 Ti 5129 : 0, 5129 section RTX A4000 4945 : 0, 4945 section RTX A5000M 4461 : 0, 4461 section RTX 3060 4070 : 0, 4070 section RTX 3060M 4012 : 0, 4012 section A16 (1 GPU) 2031 : 0, 2031 section A2 2051 : 0, 2051 section RTX 3050M Ti 2341 : 0, 2341 section RTX 3050M 2339 : 0, 2339 section RTX 3050 6GB 1898 : 0, 1898 section Titan RTX 7554 : 0, 7554 section RTX 6000 6879 : 0, 6879 section RTX 8000 Passive 5607 : 0, 5607 section RTX 2080 Ti 6853 : 0, 6853 section RTX 2080 Super 5284 : 0, 5284 section RTX 5000 4773 : 0, 4773 section RTX 2080 4977 : 0, 4977 section RTX 2070 Super 4893 : 0, 4893 section RTX 2070 5017 : 0, 5017 section RTX 2060 Super 5035 : 0, 5035 section RTX 4000 4584 : 0, 4584 section RTX 2060 KO 3376 : 0, 3376 section RTX 2060 3604 : 0, 3604 section GTX 1660 Super 3551 : 0, 3551 section T4 2887 : 0, 2887 section GTX 1660 Ti 3041 : 0, 3041 section GTX 1660 1992 : 0, 1992 section GTX 1650M 896C 1858 : 0, 1858 section GTX 1650M 1024C 1400 : 0, 1400 section T500 665 : 0, 665 section Titan Xp 5495 : 0, 5495 section GTX 1080 Ti 4877 : 0, 4877 section GTX 1080 3182 : 0, 3182 section GTX 1060 6GB 1925 : 0, 1925 section GTX 1060M 1882 : 0, 1882 section GTX 1050M Ti 1224 : 0, 1224 section P1000 839 : 0, 839 section GTX TITAN X 2665 : 0, 2665 section GTX 980 Ti 2703 : 0, 2703 section GTX 980 1965 : 0, 1965 section GTX 970 1721 : 0, 1721 section M4000 1519 : 0, 1519 section M60 (1 GPU) 1571 : 0, 1571 section GTX 960M 872 : 0, 872 section GTX 780 Ti 2776 : 0, 2776 section GTX 770 1215 : 0, 1215 section GTX 680 4GB 1274 : 0, 1274 section GTX 670 1220 : 0, 1220 section GTX 660 1146 : 0, 1146 section GTX 660 OEM 990 : 0, 990 section K2000 444 : 0, 444 section GT 630 (OEM) 185 : 0, 185 section GTX 580 1481 : 0, 1481 section GTX 560 Ti 895 : 0, 895 section GTX 480 1264 : 0, 1264 section GTX 280 462 : 0, 462 section GTX 260 421 : 0, 421 section FX 5800 296 : 0, 296 section NVS 290 9 : 0, 9 section Arise 1020 6 :active, 0, 6

section M2 Ultra (76-CU, 192GB) 8769 :active, 0, 8769 section M2 Max (38-CU, 32GB) 4641 :active, 0, 4641 section M2 Pro (19-CU, 16GB) 2374 :active, 0, 2374 section M1 Ultra (64-CU, 128GB) 8418 :active, 0, 8418 section M1 Max (24-CU, 32GB) 4496 :active, 0, 4496 section M1 Pro (16-CU, 16GB) 2329 :active, 0, 2329 section M1 (8-CU, 16GB) 759 :active, 0, 759 section Radeon 8060S (Max+ 395) 2563 :crit, 0, 2563 section Radeon 780M (Z1 Extreme) 860 :crit, 0, 860 section Radeon Graphics (7800X3D) 498 :crit, 0, 498 section Vega 8 (4750G) 511 :crit, 0, 511 section Vega 8 (3500U) 288 :crit, 0, 288 section Arc 140V GPU (16GB) 1282 :done, 0, 1282 section Arc Graphics (Ultra 9 185H) 724 :done, 0, 724 section Iris Xe Graphics (i7-1265U) 621 :done, 0, 621 section UHD Xe 32EUs 245 :done, 0, 245 section UHD 770 475 :done, 0, 475 section UHD 630 301 :done, 0, 301 section UHD P630 288 :done, 0, 288 section HD 5500 192 :done, 0, 192 section HD 4600 115 :done, 0, 115 section Orange Pi 5 Mali-G610 MP4 232 :active, 0, 232 section Samsung Mali-G72 MP18 230 :active, 0, 230

section 2x EPYC 9754 5179 :crit, 0, 5179 section 2x EPYC 9654 4092 :crit, 0, 4092 section 2x EPYC 9554 2552 :crit, 0, 2552 section 1x EPYC 9124 772 :crit, 0, 772 section 2x EPYC 7713 1418 :crit, 0, 1418 section 2x EPYC 7352 739 :crit, 0, 739 section 2x EPYC 7313 498 :crit, 0, 498 section 2x EPYC 7302 784 :crit, 0, 784 section 2x Xeon 6980P 7875 :done, 0, 7875 section 2x Xeon 6979P 8135 :done, 0, 8135 section 2x Xeon 6960P 5477 :done, 0, 5477 section 2x Platinum 8592+ 3135 :done, 0, 3135 section 2x Gold 6548N 1811 :done, 0, 1811 section 2x CPU Max 9480 2037 :done, 0, 2037 section 2x Platinum 8480+ 2162 :done, 0, 2162 section 2x Platinum 8470 2068 :done, 0, 2068 section 2x Gold 6438Y+ 1945 :done, 0, 1945 section 2x Platinum 8380 1410 :done, 0, 1410 section 2x Platinum 8358 1285 :done, 0, 1285 section 2x Platinum 8256 396 :done, 0, 396 section 2x Platinum 8153 691 :done, 0, 691 section 2x Gold 6248R 755 :done, 0, 755 section 2x Gold 6128 254 :done, 0, 254 section Phi 7210 415 :done, 0, 415 section 4x E5-4620 v4 460 :done, 0, 460 section 2x E5-2630 v4 264 :done, 0, 264 section 2x E5-2623 v4 125 :done, 0, 125 section 2x E5-2680 v3 304 :done, 0, 304 section GH200 Neoverse-V2 1323 : 0, 1323 section TR PRO 7995WX 1715 :crit, 0, 1715 section TR 3970X 463 :crit, 0, 463 section TR 1950X 273 :crit, 0, 273 section Ryzen 7900X3D 521 :crit, 0, 521 section Ryzen 7800X3D 363 :crit, 0, 363 section Ryzen 5700X3D 229 :crit, 0, 229 section FX-6100 22 :crit, 0, 22 section Athlon X2 QL-65 3 :crit, 0, 3 section Ultra 7 258V 287 :done, 0, 287 section Ultra 9 185H 317 :done, 0, 317 section i9-14900K 490 :done, 0, 490 section i7-13700K 504 :done, 0, 504 section i7-1265U 128 :done, 0, 128 section i9-11900KB 208 :done, 0, 208 section i9-10980XE 286 :done, 0, 286 section E-2288G 198 :done, 0, 198 section i7-9700 103 :done, 0, 103 section i5-9600 147 :done, 0, 147 section i7-8700K 152 :done, 0, 152 section E-2176G 201 :done, 0, 201 section i7-7700HQ 108 :done, 0, 108 section E3-1240 v5 141 :done, 0, 141 section i5-5300U 37 :done, 0, 37 section i7-4770 104 :done, 0, 104 section i7-4720HQ 80 :done, 0, 80 section N2807 7 :done, 0, 7 ```

Multi-GPU Benchmarks

Multi-GPU benchmarks are done at the largest possible grid resolution with cubic domains, and either 2x1x1, 2x2x1 or 2x2x2 of these domains together. The (percentages in round brackets) are single-GPU roofline model efficiency, and the (multiplicators in round brackets) are scaling factors relative to benchmarked single-GPU performance.

```mermaid gantt

title FluidX3D Performance [MLUPs/s] - FP32 arithmetic, (fastest of FP32/FP16S/FP16C) memory storage dateFormat X axisFormat %s %%{ init: { "gantt": { 'titleTopMargin': 42, 'topPadding': 70, 'leftPadding': 260, 'rightPadding': 5, 'sectionFontSize': 20, 'fontSize': 20, 'barHeight': 20, 'barGap': 3, 'numberSectionStyles': 2 }, 'theme': 'forest', 'themeVariables': { 'sectionBkgColor': '#99999999', 'altSectionBkgColor': '#00000000', 'titleColor': '#AFAFAF', 'textColor': '#AFAFAF', 'taskTextColor': 'black', 'taskBorderColor': '#487E3A' } } }%%

section 8x MI300X 204924 :crit, 0, 204924 section 4x MI300X 109546 :crit, 0, 109546 section 2x MI300X 61053 :crit, 0, 61053 section 1x MI300X 41327 :crit, 0, 41327

section 4x MI250 (8 GCD) 53521 :crit, 0, 53521 section 2x MI250 (4 GCD) 29627 :crit, 0, 29627 section 1x MI250 (2 GCD 17338 :crit, 0, 17338 section 1x MI250 (1 GCD) 9030 :crit, 0, 9030

section 32x MI210 GigaIO 50952 :crit, 0, 50952 section 24x MI210 GigaIO 45033 :crit, 0, 45033 section 16x MI210 GigaIO 37922 :crit, 0, 37922 section 8x MI210 GigaIO 27996 :crit, 0, 27996 section 4x MI210 GigaIO 17232 :crit, 0, 17232 section 2x MI210 GigaIO 13539 :crit, 0, 13539 section 1x MI210 GigaIO 9105 :crit, 0, 9105

section 4x MI210 31408 :crit, 0, 31408 section 2x MI210 16156 :crit, 0, 16156 section 1x MI210 8757 :crit, 0, 8757

section 3x MI50 + 1x A100 40GB 22759 :active,crit, 0, 22759 section 3x MI50 32GB 21693 :crit, 0, 21693 section 2x MI50 32GB 14484 :crit, 0, 14484 section 1x MI50 32GB 8477 :crit, 0, 8477

section 8x Radeon VII 30826 :crit, 0, 30826 section 4x Radeon VII 24273 :crit, 0, 24273 section 2x Radeon VII 15591 :crit, 0, 15591 section 1x Radeon VII 7778 :crit, 0, 7778

section 1x Radeon Pro Duo (2 GPUs) 3310 :crit, 0, 3310 section 1x R9 295X2 (2 GPUs) 2428 :crit, 0, 2428 section 1x HD 7990 (2 GPUs) 2314 :crit, 0, 2314 section 1x HD 6990 (2 GPUs) 344 :crit, 0, 344 section 1x HD 5970 (2 GPUs) 360 :crit, 0, 360

section 4x GPU Max 1100 22777 :done, 0, 22777 section 2x GPU Max 1100 11815 :done, 0, 11815 section 1x GPU Max 1100 6209 :done, 0, 6209

section 8x B200 SXM6 180GB 219300 : 0, 219300 section 4x B200 SXM6 180GB 147446 : 0, 147446 section 2x B200 SXM6 180GB 85077 : 0, 85077 section 1x B200 SXM6 180GB 55609 : 0, 55609

section 8x H200 SXM5 141GB 157743 : 0, 157743 section 4x H200 SXM5 141GB 96056 : 0, 96056 section 2x H200 SXM5 141GB 57070 : 0, 57070 section 1x H200 SXM5 141GB 36610 : 0, 36610

section 4x H100 NVL 82122 : 0, 82122 section 2x H100 NVL 49958 : 0, 49958 section 1x H100 NVL 32922 : 0, 32922

section 4x H100 SXM5 80GB HBM3 78462 : 0, 78462 section 2x H100 SXM5 80GB HBM3 46189 : 0, 46189 section 1x H100 SXM5 80GB HBM3 28522 : 0, 28522

section 4x A100 PCIe 80GB 52056 : 0, 52056 section 2x A100 PCIe 80GB 27165 : 0, 27165 section 1x A100 PCIe 80GB 17896 : 0, 17896

section 4x PG506-243/242 41088 : 0, 41088 section 2x PG506-243/242 24168 : 0, 24168 section 1x PG506-243/242 15654 : 0, 15654

section 8x A100 SXM4 40GB 72965 : 0, 72965 section 4x A100 SXM4 40GB 42400 : 0, 42400 section 2x A100 SXM4 40GB 23707 : 0, 23707 section 1x A100 SXM4 40GB 15917 : 0, 15917

section 4x V100 SXM2 32GB 26527 : 0, 26527 section 2x V100 SXM2 32GB 15469 : 0, 15469 section 1x V100 SXM2 32GB 8947 : 0, 8947

section 3x K40m + 1x Titan Xp 5174 : 0, 5174 section 2x Tesla K40m 3300 : 0, 3300 section 1x Tesla K40m 1868 : 0, 1868

section 1x Tesla K80 (2 GPUs) 3448 : 0, 3448 section 1x Tesla K80 (1 GPU) 1642 : 0, 1642

section 2x L40S 13640 : 0, 13640 section 1x L40S 7669 : 0, 7669

section 2x L40 14164 : 0, 14164 section 1x L40 7945 : 0, 7945

section 8x RTX A6000 40063 : 0, 40063 section 4x RTX A6000 27915 : 0, 27915 section 2x RTX A6000 15026 : 0, 15026 section 1x RTX A6000 8814 : 0, 8814

section 4x A16 (16 GPUs) 22451 : 0, 22451 section 2x A16 (8 GPUs) 11777 : 0, 11777 section 1x A16 (4 GPUs) 6348 : 0, 6348 section 1x A16 (2 GPUs) 3475 : 0, 3475 section 1x A16 (1 GPU) 2031 : 0, 2031

section 2x A2 3539 : 0, 3539 section 1x A2 2051 : 0, 2051

section 2x Quadro RTX 8000 Pa. 10214 : 0, 10214 section 1x Quadro RTX 8000 Pa. 5607 : 0, 5607

section 7x 2080 Ti + 1x A100 40GB 33857 : 0, 33857 section 4x GeForce RTX 2080 Ti 18598 : 0, 18598 section 2x GeForce RTX 2080 Ti 10922 : 0, 10922 section 1x GeForce RTX 2080 Ti 6853 : 0, 6853

section 1x GTX 690 (2 GPUs) 920 : 0, 920

section 2x Arc A770 8745 :done, 0, 8745 section 1x Arc A770 4568 :done, 0, 4568

section 1x A100 + 1x P100 + 2x A2 + 3x MI50 + 1x A770 17296 :active,crit, 0, 17296 section 1x 7700 XT + 1x B580 + 1x Titan Xp 8358 :active,crit, 0, 8358 section 1x A770 + 1x Titan Xp 8380 :active,done, 0, 8380 ```

FAQs

General

• How to learn using FluidX3D?

  
  Follow the FluidX3D Documentation!
  
  
• What physical model does FluidX3D use?

  
  FluidX3D implements the lattice Boltzmann method, a type of direct numerical simulation (DNS), the most accurate type of fluid simulation, but also the most computationally challenging. Optional extension models include volume force (Guo forcing), free surface (volume-of-fluid and PLIC), a temperature model and Smagorinsky-Lilly subgrid turbulence model.
  
  
• FluidX3D only uses FP32 or even FP32/FP16, in contrast to FP64. Are simulation results physically accurate?

  
  Yes, in all but extreme edge cases. The code has been specially optimized to minimize arithmetic round-off errors and make the most out of lower precision. With these optimizations, accuracy in most cases is indistinguishable from FP64 double-precision, even with FP32/FP16 mixed-precision. Details can be found in this paper.
  
  
• Compared to the benchmark numbers stated here, efficiency seems much lower but performance is slightly better for most devices. How can this be?

  
  In that paper, the One-Step-Pull swap algorithm is implemented, using only misaligned reads and coalesced writes. On almost all GPUs, the performance penalty for misaligned writes is much larger than for misaligned reads, and sometimes there is almost no penalty for misaligned reads at all. Because of this, One-Step-Pull runs at peak bandwidth and thus peak efficiency.
  Here, a different swap algorithm termed Esoteric-Pull is used, a type of in-place streaming. This makes the LBM require much less memory (93 vs. 169 (FP32/FP32) or 55 vs. 93 (FP32/FP16) Bytes/cell for D3Q19), and also less memory bandwidth (153 vs. 171 (FP32/FP32) or 77 vs. 95 (FP32/FP16) Bytes/cell per time step for D3Q19) due to so-called implicit bounce-back boundaries. However memory access now is half coalesced and half misaligned for both reads and writes, so memory access efficiency is lower. For overall performance, these two effects approximately cancel out. The benefit of Esoteric-Pull - being able to simulate domains twice as large with the same amount of memory - clearly outweights the cost of slightly lower memory access efficiency, especially since performance is not reduced overall.
  
  
• Why don't you use CUDA? Wouldn't that be more efficient?

  
  No, that is a wrong myth. OpenCL is exactly as efficient as CUDA on Nvidia GPUs if optimized properly. Here I did roofline model and analyzed OpenCL performance on various hardware. OpenCL efficiency on modern Nvidia GPUs can be 100% with the right memory access pattern, so CUDA can't possibly be any more efficient. Without any performance advantage, there is no reason to use proprietary CUDA over OpenCL, since OpenCL is compatible with a lot more hardware.
  
  
• Why no multi-relaxation-time (MRT) collision operator?

  
  The idea of MRT is to linearly transform the DDFs into "moment space" by matrix multiplication and relax these moments individually, promising better stability and accuracy. In practice, in the vast majority of cases, it has zero or even negative effects on stability and accuracy, and simple SRT is much superior. Apart from the kinematic shear viscosity and conserved terms, the remaining moments are non-physical quantities and their tuning is a blackbox. Although MRT can be implemented in an efficient manner with only a single matrix-vector multiplication in registers, leading to identical performance compared to SRT by remaining bandwidth-bound, storing the matrices vastly elongates and over-complicates the code for no real benefit.
  
  

Hardware

• Can FluidX3D run on multiple GPUs at the same time?

  
  Yes. The simulation grid is then split in domains, one for each GPU (domain decomposition method). The GPUs essentially pool their memory, enabling much larger grid resolution and higher performance. Rendering is parallelized across multiple GPUs as well; each GPU renders its own domain with a 3D offset, then rendered frames from all GPUs are overlayed with their z-buffers. Communication between domains is done over PCIe, so no SLI/Crossfire/NVLink/InfinityFabric is required. All GPUs must however be installed in the same node (PC/laptop/server). Even unholy combinations of AMD+Intel+Nvidia GPUs will work, although it is recommended to only use GPUs with similar memory capacity and bandwidth together. Using a fast gaming GPU and slow integrated GPU together would only decrease performance due to communication overhead.
  
  
• I'm on a budget and have only a cheap computer. Can I run FluidX3D on my toaster PC/laptop?

  
  Absolutely. Today even the most inexpensive hardware, like integrated GPUs or entry-level gaming GPUs, support OpenCL. You might be a bit more limited on memory capacity and grid resolution, but you should be good to go. I've tested FluidX3D on very old and inexpensive hardware and even on my Samsung S9+ smartphone, and it runs just fine, although admittedly a bit slower.
  
  
• I don't have an expensive workstation GPU, but only a gaming GPU. Will performance suffer?

  
  No. Efficiency on gaming GPUs is exactly as good as on their "professional"/workstation counterparts. Performance often is even better as gaming GPUs have higher boost clocks.
  
  
• Do I need a GPU with ECC memory?

  
  No. Gaming GPUs work just fine. Some Nvidia GPUs automatically reduce memory clocks for compute applications to almost entirely eliminate memory errors.
  
  
• My GPU does not support CUDA. Can I still use FluidX3D?

  
  Yes. FluidX3D uses OpenCL 1.2 and not CUDA, so it runs on any GPU from any vendor since around 2012.
  
  
• I don't have a dedicated graphics card at all. Can I still run FluidX3D on my PC/laptop?

  
  Yes. FluidX3D also runs on all integrated GPUs since around 2012, and also on CPUs.
  
  
• I need more memory than my GPU can offer. Can I run FluidX3D on my CPU as well?

  
  Yes. You only need to install the Intel OpenCL CPU Runtime.
  
  
• In the benchmarks you list some very expensive hardware. How do you get access to that?

  
  As a PhD candidate in computational physics, I used FluidX3D for my research, so I had access to BZHPC, SuperMUC-NG and JSC JURECA-DC supercomputers.
  
  

Graphics

• I don't have an RTX/DXR GPU that supports raytracing. Can I still use raytracing graphics in FluidX3D?

  
  Yes, and at full performance. FluidX3D does not use a bounding volume hierarchy (BVH) to accelerate raytracing, but fast ray-grid traversal instead, implemented directly in OpenCL C. This is much faster than BVH for moving isosurfaces in the LBM grid (~N vs. ~N²+log(N) runtime; LBM itself is ~N³), and it does not require any dedicated raytracing hardware. Raytracing in FluidX3D runs on any GPU that supports OpenCL 1.2.
  
  
• I have a datacenter/mining GPU without any video output or graphics hardware. Can FluidX3D still render simulation results?

  
  Yes. FluidX3D does all rendering (rasterization and raytracing) in OpenCL C, so no display output and no graphics features like OpenGL/Vulkan/DirectX are required. Rendering is just another form of compute after all. Rendered frames are passed to the CPU over PCIe and then the CPU can either draw them on screen through dedicated/integrated graphics or write them to the hard drive.
  
  
• I'm running FluidX3D on a remote (super-)computer and only have an SSH terminal. Can I still use graphics somehow?

  
  Yes, either directly as interactive ASCII graphics in the terminal or by storing rendered frames on the hard drive and then copying them over via `scp -r user@server.url:"~/path/to/images/folder" .`.
  
  

Licensing

• I want to learn about programming/software/physics/engineering. Can I use FluidX3D for free?

  
  Yes. Anyone can use FluidX3D for free for public research, education or personal use. Use by scientists, students and hobbyists is free of charge and well encouraged.
  
  
• I am a scientist/teacher with a paid position at a public institution. Can I use FluidX3D for my research/teaching?

  
  Yes, you can use FluidX3D free of charge. This is considered research/education, not commercial use. To give credit, the references listed below should be cited. If you publish data/results generated by altered source versions, the altered source code must be published as well.
  
  
• I work at a company in CFD/consulting/R&D or related fields. Can I use FluidX3D commercially?

  
  No. Commercial use is not allowed with the current license.
  
  
• Is FluidX3D open-source?

  
  No. "Open-source" as a technical term is defined as freely available without any restriction on use, but I am not comfortable with that. I have written FluidX3D in my spare time and no one should milk it for profits while I remain uncompensated, especially considering what other CFD software sells for. The technical term for the type of license I choose is "source-available no-cost non-commercial". The source code is freely available, and you are free to use, to alter and to redistribute it, as long as you do not sell it or make a profit from derived products/services, and as long as you do not use it for any military purposes (see the license for details).
  
  
• Will FluidX3D at some point be available with a commercial license?

  
  Maybe I will add the option for a second, commercial license later on. If you are interested in commercial use, let me know. For non-commercial use in science and education, FluidX3D is and will always be free.
  
  

External Code/Libraries/Images used in FluidX3D

• OpenCL-Headers and C++ Wrapper for GPU parallelization (Khronos Group)
• Win32 API for interactive graphics in Windows (Microsoft)
• X11/Xlib for interactive graphics in Linux (The Open Group)
• marching-cubes tables for isosurface generation on GPU (Paul Bourke)
• src/lodepng.cpp and src/lodepng.hpp for .png encoding and decoding (Lode Vandevenne)
• SimplexNoise class in src/utilities.hpp for generating continuous noise in 2D/3D/4D space (Stefan Gustavson)
• skybox/skybox8k.png for free surface raytracing (HDRI Hub)

References

• Lehmann, M.: Computational study of microplastic transport at the water-air interface with a memory-optimized lattice Boltzmann method. PhD thesis, (2023)
• Lehmann, M.: Esoteric Pull and Esoteric Push: Two Simple In-Place Streaming Schemes for the Lattice Boltzmann Method on GPUs. Computation, 10, 92, (2022)
• Lehmann, M., Krause, M., Amati, G., Sega, M., Harting, J. and Gekle, S.: Accuracy and performance of the lattice Boltzmann method with 64-bit, 32-bit, and customized 16-bit number formats. Phys. Rev. E 106, 015308, (2022)
• Lehmann, M.: Combined scientific CFD simulation and interactive raytracing with OpenCL. IWOCL'22: International Workshop on OpenCL, 3, 1-2, (2022)
• Lehmann, M., Oehlschlägel, L.M., Häusl, F., Held, A. and Gekle, S.: Ejection of marine microplastics by raindrops: a computational and experimental study. Micropl.&Nanopl. 1, 18, (2021)
• Lehmann, M.: High Performance Free Surface LBM on GPUs. Master's thesis, (2019)
• Lehmann, M. and Gekle, S.: Analytic Solution to the Piecewise Linear Interface Construction Problem and Its Application in Curvature Calculation for Volume-of-Fluid Simulation Codes. Computation, 10, 21, (2022)

Contact

• FluidX3D is solo-developed and maintained by Dr. Moritz Lehmann.
• For any questions, feedback or other inquiries, contact me at dr.moritz.lehmann@gmail.com.
• Updates are posted on Mastodon via @ProjectPhysX/#FluidX3D and on YouTube.

Support

I'm developing FluidX3D in my spare time, to make computational fluid dynamics lightning fast, accessible on all hardware, and free for everyone. - You can support FluidX3D by reporting any bugs or things that don't work in the issues. I'm welcoming feedback! - If you like FluidX3D, share it with friends and colleagues. Spread the word that CFD is now lightning fast, accessible and free. - If you want to support FluidX3D financially, you can sponsor me on GitHub or buy me a coffee. Thank you!