rTLsDeep: An R Package for post-hurricane damage severity classification at the individual tree level using terrestrial laser scanning and deep learning.

Authors: Carine Klauberg, Carlos Alberto Silva, Ricardo Dalagnol, Matheus Ferreira, Danilo Romeu Farias de Souza, Luiz Guilherme Almeida Nogueira, Eben Broadbent, Caio Hamamura and Jason Vogel.

The rTLSDeep package provides options for i) rotating and deriving 2D images from TLS 3D point clouds, ii) calibrating and validating convolutional neural network (CNN) architectures and iii) predicting post-hurricane damage severity at the individual tree level

Getting Started

Install R, Git and Rtools40

1. R >= 4.0.0: https://www.r-project.org/
2. Rtools >= 4 (windows): https://cran.r-project.org/bin/windows/Rtools/
3. Git: https://git-scm.com/
4. tensorflow (python): https://doi.org/10.5281/zenodo.3929709
5. numpy (python)
6. scipy (python)
7. pillow (python - recommended)

rTLsDeep installation

```r

The CRAN version:

install.packages("rTLsDeep")

The development version:

install.packages("remotes")

library(remotes) install_github("https://github.com/carlos-alberto-silva/rTLsDeep", dependencies = TRUE) ```

Getting Started

Loading rTLsDeep and other required packages

```r

get pacman

install.packages("pacman")

load pcaman and all packages

library(pacman) p_load(rTLsDeep,lidR,rgl,ggplot2,rgl,keras,reticulate,compiler,terra) ```

TLS data processing

Loading and visualizing TLS dataset

```r

Path to las file

lasfile <- system.file("extdata", "tree_c1.laz", package="rTLsDeep")

Reading las file

las<-readLAS(lasfile)

plotting las file in 3D

plot(las, bg="white") rgl::axes3d(c("x+", "y-", "z-"), col="black") rgl::grid3d(side=c('x+', 'y-', 'z'), col="gray") ```

Rotating TLS-derived 3d point cloud

```r

Rotating around the x-axis

las<-tlsrotate3d(las,theta=120, by="x", scale=TRUE)

Rotating around the y-axis

las<-tlsrotate3d(las,theta=120, by="y", scale=TRUE)

Rotating around the z-axis

las<-tlsrotate3d(las,theta=120, by="z", scale=TRUE) ```

Capturing 2D grid snapshot

```r

Set output dir for downloading the example dataset files

outdir=getwd()

downloading zip file

download.file("https://github.com/carlos-alberto-silva/rTLsDeep/raw/main/readme/lazfiles.zip",destfile=file.path(outdir, "lazfiles.zip"))

unzip file

unzip(file.path(outdir,"laz_files.zip"))

Reading las file for each post-hurricane individual tree-level damage classes

treec1<-readLAS(file.path(outdir,"laz","Treec1.laz")) treec2<-readLAS(file.path(outdir,"laz","Treec2.laz")) treec3<-readLAS(file.path(outdir,"laz","Treec3.laz")) treec4<-readLAS(file.path(outdir,"laz","Treec4.laz")) treec5<-readLAS(file.path(outdir,"laz","Treec5.laz")) treec6<-readLAS(file.path(outdir,"laz","Treec6.laz"))

Defining the func parameter

func = ~list(Z = max(Z)) # plot by height

computing 2D grid snapshot

gtreec1<-getTLS2D(treec1, res=0.05, by="xz", func = func, scale=TRUE) gtreec2<-getTLS2D(treec2, res=0.05, by="xz", func = func, scale=TRUE) gtreec3<-getTLS2D(treec3, res=0.05, by="xz", func = func, scale=TRUE) gtreec4<-getTLS2D(treec4, res=0.05, by="xz", func = func, scale=TRUE) gtreec5<-getTLS2D(treec5, res=0.05, by="xz", func = func, scale=TRUE) gtreec6<-getTLS2D(treec6, res=0.05, by="xz", func = func, scale=TRUE)

Visualizing 2D grid snapshot

par(mfrow=c(2,3)) plot(gtreec1, col=viridis::viridis(100),axes=FALSE, xlab="",ylab="", ylim=c(0,30), main="C1",cex=2) plot(gtreec2, col=viridis::viridis(100),axes=FALSE, xlab="",ylab="", ylim=c(0,30), main="C2",cex=2) plot(gtreec3, col=viridis::viridis(100),axes=FALSE, xlab="",ylab="", ylim=c(0,30), main="C3",cex=2) plot(gtreec4, col=viridis::viridis(100),axes=FALSE, xlab="",ylab="", ylim=c(0,30), main="C4",cex=2) plot(gtreec5, col=viridis::viridis(100),axes=FALSE, xlab="",ylab="", ylim=c(0,30), main="C5",cex=2) plot(gtreec6, col=viridis::viridis(100),axes=FALSE, xlab="",ylab="", ylim=c(0,30), main="C6",cex=2) ```

Post-hurricane individual tree-level damage classification using deep learning

Selecting deep learning model properties

```r

Exporting 2D grids

Creating train/test datasets for C1 and C2 only

classes = c('C1', 'C2') targets = c('train', 'validation')

Create folders

folderstocreate = file.path(targets, rep(classes, each=length(targets))) sapply(folderstocreate, dir.create, recursive=T, showWarnings=FALSE)

Calculate rotations by 15 degrees

rotations = c(seq(0, 89, 15), seq(180, 269, 15)) n_images = length(rotations) * 2 # We will use xz and yz from TLS

Get random training samples

indices = seqlen(nimages) valsamples = sample(indices, size=nimages * 0.25)

Image parameters

imgwidth = 256 imgheight = 256 resolution = 0.05 max_height = 30

createImage = function(raster, filepath, width, height) { png(paste0(filepath, '.png'), units="px", width=imgwidth, height=imgheight) par(mar=c(0,0,0,0)) terra::image(raster, col=viridis::viridis(100), axes=FALSE, ylim=c(-resolution,max_height-resolution)) dev.off() }

ii = 1 func = ~list(Z = max(Z)) # plot by height

Create images rotating images

for (rotation in rotations) { trainorvals = targets[(c(ii, ii+1) %in% val_samples) + 1]

for (current_class in classes) {
    tree = tlsrotate3d(get(paste0('tree_', tolower(current_class))), theta=rotation, by="z", scale=TRUE)

    raster = getTLS2D(tree, res=resolution, by="xz", func = func, scale=TRUE)
    createImage(raster, file.path(train_or_vals[1], current_class, ii), width, height)

    raster = getTLS2D(tree, res=resolution, by="yz", func = func, scale=TRUE)
    createImage(raster, file.path(train_or_vals[2], current_class, ii + 1), width, height)
}
ii = ii + 2

}

Set directory to tensorflow (python environment)

This is required if running deep learning local computer with GPU

Guide to install here: https://doi.org/10.5281/zenodo.3929709

tensorflow_dir = NA

define model type

model_type = "simple"

model_type = "vgg"

model_type = "inception"

model_type = "resnet"

model_type = "densenet"

model_type = "efficientnet"

trainimagefilespath = 'train' testimagefilespath = 'validation'

Image and model properties

imgwidth <- 256 imgheight <- 256 classlisttrain = unique(list.files(trainimagefilespath)) classlisttest = unique(list.files(testimagefilespath)) lrrate = 0.0001 targetsize <- c(imgwidth, imgheight) channels <- 4 batch_size = 8L epochs = 20L

get model

model = getdlmodel(modeltype=modeltype, imgwidth=imgwidth, imgheight=imgheight, channels=channels, lrrate = lrrate, tensorflowdir = tensorflowdir, classlist = classlist_train)

```

Model calibration

```r weightsfname = fitdlmodel(model = model, traininputpath = trainimagefilespath, testinputpath = testimagefilespath, targetsize = targetsize, batchsize = batchsize, classlist = classlisttrain, epochs = epochs, lrrate = lrrate)

```

Predicting post-hurricane damage at the tree-level

r tree_damage<-predict_treedamage(model = model, input_file_path = test_image_files_path, weights = weights_fname, target_size = c(256,256), class_list=class_list_test, batch_size = batch_size)

Confusion matrix

```r

Get damage classes for validation datasets

testclasses<-getvalidationclasses(filepath=testimagefiles_path)

Calculate confusion matrix

cm = confmatrixtreedamage(predictclass = treedamage, testclasses=testclasses, classlist = classlisttest)

Plot confusion matrix

gcmplot_vgg<-gcmplot(cm, colors=c(low="white", high="#009194"), title="densenet") ```

Finding the best rotation

```

Find the best angle and perform rotation

(bestanglec2 = getbestangle(treec2)) rotatedc2 = tlsrotate3d(treec2, theta = bestangle_c2)

computing 2D grid snapshot

gtreerotatedc2<-getTLS2D(rotated_c2, res=0.05, by="xz", func = func, scale=TRUE)

Visualizing 2D grid snapshot

par(mfrow=c(1,2)) plot(gtreec2, col=viridis::viridis(100),axes=FALSE, legend=FALSE, xlab="",ylab="", ylim=c(0,30), main="C2",cex=2) plot(gtreerotatedc2, col=viridis::viridis(100),axes=FALSE, legend=FALSE, xlab="",ylab="", ylim=c(0,30), main=gettextf("C2 (best %s°)", round(bestanglec2,0)),cex=2) ``` <img src="https://github.com/carlos-alberto-silva/rTLsDeep/blob/main/readme/treerotation.png" style="width:50%;">

Working example using Google Colab:

https://colab.research.google.com/drive/1YnvIca1FtHqIYwWKmp5zoPOz1Zonm7NR?usp=sharing

References

R Core Team. (2021). R: A Language and Environment for Statistical Computing; R Core Team: Vienna, Austria. https://www.r-project.org/

Acknowledgements

We gratefully acknowledge funding from NIFA Award # 2020-67030-30714.

Reporting Issues

Please report any issue regarding the rTLsDeep package to Dr. Carlos A. Silva (c.silva@ufl.edu; maintainer)

Citing rTLsDeep application

Klauberg, C., Vogel, J., Dalagnol, R., Ferreira, M., Broadbent,E.N.; Hamamura, C., Silva, C.A. Post-hurricane damage severity classification at the individual tree level using terrestrial laser scanning and deep learning. Remote Sensing. in review

Klauberg, C., Vogel, J., Dalagnol, R., Ferreira, M., Broadbent,E.N.; Hamamura, C., Souza, D. R. F, Silva, Nogueira, L. G. A., C.A. rTLsDeep: An R Package for post-hurricane damage severity classification at the individual tree level using terrestrial laser scanning and deep learning. Version 0.0.1, accessed on December. 30 2022, available at: https://github.com/carlos-alberto-silva/rTLsDeep

Disclaimer

rTLsDeep has been developed using R (R Core Team 2022), and it comes with no guarantee, expressed or implied, and the authors hold no responsibility for its use or reliability of its outputs.