cuDFNsys

cuDFNsys is an open-source CUDA library (under the GPL license) for DFN generations. It is also able to simulate flow (steady-state) and particle transport in DFNs, based on the mixed hybrid finite element method and particle tracking algorithm. cuDFNsys contains around 10 000 lines of codes (counted on June 29th, 2023).

cuDFNsys has a simple GUI and can run on Ubuntu or Windows WSL. See manual for more details.

cuDFNsys is also an easy-implemented, object-oriented CUDA C++ library. The simulation is performed just by establishing CUDA C++ classes, defining member variables and calling member functions. The results are stored in a HDF5 format. At runtime, the fracture, mesh or flow data can be accessed by visting member variables of classes.

Figure 1. Dispersion in a DFN with 300 000 particles. The movement of particles is purely driven by the advection. Left: the DFN and hydraulic head field. Right: the particle spreading.

Figure 2. Diffusion-dominated dispersion in a DFN with 300 000 particles. The Peclet number is 0.01. The length scale in the Peclet number is the mean value of fracture sizes. Left: the DFN and hydraulic head field. Right: the particle spreading. The particles initialize at a plane with z = 0.

Authors

• Tingchang YIN, Westlake University & Zhejiang University, China, yintingchang@foxmail.com

• Sergio GALINDO-TORRES, Westlake University, China, s.torres@westlake.edu.cn

Terms

cuDFNsys - simulating flow and transport in 3D fracture networks Copyright (C) 2022, Tingchang YIN, Sergio GALINDO-TORRES

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.

Relevant publications

• T Yin, T Man, Ling Li, SA Galindo-Torres. Finite-Size Scaling for the Permeability of Discrete Fracture Networks (2023). Geophysical Research Letters, 50, e2022GL100837
• T Yin, T Man, SA Galindo-Torres. Universal scaling solution for the connectivity of discrete fracture networks (2022). Physica A: Statistical Mechanics and its Applications 599, 127495

Prerequisites

cuDFNsys should be installed and run on Ubuntu.

cuDFNsys relies on several open packages: OpenMP, CUDA, Eigen, Gmsh, UMFPACK and HDF5. These dependencies can be installed by using sudo apt-get install packageName-dev.

The following is the Ubuntu command lines to install the relying packages: ```

run the following command one by one to install dependencies

I have tested them on Ubuntu 23.04

first update the package repository information

cd ~ sudo apt-get update sudo apt-get install build-essential sudo apt-get install gfortran sudo apt-get install cmake

note that the version of cmake should be 3.10 or higher

if the version is not satisfied, try to install cmake from source

cuda

sudo apt-get install nvidia-cuda-dev sudo apt-get install nvidia-cuda-toolkit

openMP

sudo apt-get install libomp-dev

install Eigen

sudo apt-get install libeigen3-dev

create link (optional)

sudo ln -s /usr/include/eigen3/Eigen/ /usr/include/Eigen

install blas, lapack and umfpack (suitesparse)

sudo apt-get install libblas-dev liblapack-dev sudo apt-get install libsuitesparse-dev

install HDF5

sudo apt-get install libhdf5-dev

install occt for gmsh

sudo apt-get install libocct-foundation-dev libocct-data-exchange-dev

install fltk for gmsh

sudo apt-get install libfltk1.3-dev

now install gmsh

sudo apt-get install libgmsh-dev ```

Building cuDFNsys

After the installation of dependencies (by sudo apt-get install, these relying packages are installed at default locations), and the cuDFNsys library can be installed by the following steps: ``` git clone https://github.com/qq1012510777/cuDFNsys.git cd ~/cuDFNsys/lib

here I put the cuDFNsys source code under HOME

Open `SetPaths.cmake`, one will see

NVCC directory

SET(NVCC_PATH /usr/lib/nvidia-cuda-toolkit/bin/nvcc)

Eigen include directory

SET(EIGENINCLUDESEARCH_PATH /usr/include)

Gmsh include directory

SET(GMSHINCLUDESEARCH_PATH /usr/include)

Gmsh lib directory

SET(GMSHLIBRARYSEARCHPATH /usr/lib/x8664-linux-gnu)

umfpack include directory

SET(UMFPACKINCLUDESEARCH_PATH /usr/include/suitesparse)

umfpack lib directory

SET(UMFPACKLIBRARYSEARCHPATH /usr/lib/x8664-linux-gnu) ``` The above script of CMake shows that paths of the dependencies in a computer with Ubuntu 23.04. Generally these paths do not need to be changed.

But, if one install the dependencies in different paths, change these paths, e.g., /usr/include to the path of the packages in your computer, e.g., /path/in/your/computer.

Now, one can compile cuDFNsys by: cd ~/cuDFNsys/lib mkdir build cd build cmake -DCMAKE_CUDA_ARCHITECTURES=60 .. make cd .. If these commands are finished without errors, then the file libcuDFNsys.a should appear in ~/cuDFNsys/lib.

More details about the installation of the relying packages:

• Installation of CUDA

• Installation of Gmsh and occt

• Installation of Eigen

• Installation of UMFPACK

If you get the error after run make /usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’ it can be resolved by replacing the current nvcc to a higher version 11.6 or higher.

Compile a quickstart example

By cd ~/cuDFNsys/QuickStartGuide, a Makefile can be seen there. Open it, and one can see ```

NVCC path

NVCC=/usr/lib/nvidia-cuda-toolkit/bin/nvcc

include paths for headers

cuDFNsysIncludePath=$(HOME)/cuDFNsys/include Hdf5IncludePath=/usr/include/hdf5/serial GmshIncludePath=usr/include EigenIncludePath=/usr/include UmfpackIncludePath=/usr/include/suitesparse

library paths

cuDFNsysLibraryPath=$(HOME)/cuDFNsys/lib GmshLibraryPath=/usr/lib/x8664-linux-gnu UmfpackLibraryPath=/usr/lib/x8664-linux-gnu Hdf5LibraryPath=/usr/lib/x86_64-linux-gnu/hdf5/serial Change the paths of the headers and libraries (if necessary), and compile the QuickStartGuide.cu just by make ``` and seversal executable files can be generated.

After the compilation of QuickStartGuide, run one executable file by ./QuickStartGuide If errors happen, e.g., error while loading shared libraries: libgmsh.so: cannot open shared object file: No such file or directory, just add two enviromental variables to ~/.bashrc as follows: vim ~/.bashrc add the following content to ~/.bashrc ``` export LDLIBRARYPATH=path-to-gmsh-library:$LDLIBRARYPATH export LIBRARYPATH=path-to-gmsh-library:$LIBRARYPATH

change path-to-gmsh-library to the dynamic gmsh library in the computer

then update `~/.bashrc` by source ~/.bashrc `` This quickstart example can be run by./QuickStartGuide`, where a DFN is generated in which the flow is solved by a mixed hybrid finite element method and the particle tracking is implement for a not-very-long time.

cuDFNsys runs the flow and transport in DFNs by four cuda c++ classes, and the DFN model and mesh and flow data are reproducable, for example, to run more steps for particle tracking in one DFN, please see codes (i.e., ReRunPT.cpp`) and run the corresponding executable file. More details are in manual.

Visualization

After simulation, cuDFNsys outputs .h5 and .m (and/or .py) files, and you can visualize them by the generated .m or .py file.

The visulization with MATLAB is simple, you just need a MATLAB installed, and run the .m file.

With Python, you need to install the mayavi Python package. mayavi is a very good visualization engine. The Matplotlib package is not good at 3D visualizations, see Why my 3D plot doesn’t look right at certain viewing angles.

Installation of mayavi can be done by the following commands (I've test them in Ubuntu 23.04): ``` sudo apt install python3-h5py python3-vtk9 python3-pip python3-pyqt5 python3-traits python3-traitsui python3-numpy python3-matplotlib python3-setuptools python3-pyqt5.qtopengl

Please note that package names and dependencies may change over time, so make sure to check for any updates or changes in package names specific to your Ubuntu version.

sudo apt-get install mayavi2 `` To run python visualization script, just runpython3 NameOfScript.py`.

If you get the error The following packages have unmet dependencies: python3-vtk9 : Breaks: python3-vtk7 but 7.1.1+dfsg2-10.1build1 is to be installed E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages. it can be resolved by sudo apt remove python3-vtk9 sudo apt-get install python3-vtk7

Manual

Manual for cuDFNsys is here.

Now, cuDFNsys has a simple GUI, which rquires the cuDFNsys library to be compiled. Also, the GUI can visualize the DFN, mesh, flow, and PT both in Python Matplotlib and Mayavi (required it to be installed). See Manual for more details about how to run a cuDFNsys GUI.

Directories

QuickStartGuide: a quickstart guide CUDA example to show how to do simulations with cuDFNsys functions.

Modules: cmake script to find packages that cuDFNsys relies on in a Ubuntu system

include: header files containing declarations of functions/classes

src: source files containing definitions of functions/classes

SomeApplications: these are all about my personal examples (user's interfaces), with different purposes, meanwhile showing that how to call cuDFNsys functions, with make and other tools.