Geostatistical tools container

This repository contains a docker image for geostatistical data processing using standardized input and outputs. This is based on the template for a generic containerized Python tool.

Currently, there are three tools implemented:

1. Geostatistical variogram estimation
2. Kriging interpolation
3. Geostatistical simulation

In order to run an interpolation or simulation,you need to provide a valid variogram first. The variogram tool is accompanied by extensive plotting to validate its geospatial robustness. Variograms can be calculated for spatially distributed samples, or spatially dense fields. You need to provide either of both inputs. If a field is used, it is highly recommended to subsample it instead of resolving all coordinates into a sample. The tool defaults to a maximum sample size of 1000, which can be disabled by setting the sample_size parameter to -1.

To identify the spatial coordinates, the tool will look for standard column names, namely: x, y, lon, lat, longitude, latitude. It is recommended to use projected data, but in case of field information, the tool will reproject to the most likely UTM zone. Samples are never reprojected.

Samples with more than one non-spatial column will be estimated as cross-variograms. Fields with more than one variable will yield multiple variograms. If many variograms are estimated, the kriging and simulation tools can iterate over many inputs.

Parameters

The tables below summarize the availble parameters for each of the tools.

Variogram

| Parameter Name | Description | Data Type | |---|---|---| | n_lags | Number of separating distance lag classes. Is ignored for binfuncs [fd, sturges, scott, doane, sqrt] | integer | | `binfunc| Function to group the distance matrix into lag classes. | enum | |model| Interpretive theoretical variogram model function to model the covariance | enum | |estimator| Semi-variance estimation method to calculate the empirical variogram | enum | |maxlag| Can be 'median', 'mean', a number < 1 for a ratio of maximum separating distance or a number > 1 for an absolute distance | string | |fitmethod| | enum | |usenugget| Enable the nugget parameter. Defaults to False, which will set the nugget parameter to 0. | bool | |fitrange` | Only valid if fitmethod='manual'. The variogram effective range. | float | | fit_sill | Only valid if fitmethod='manual'. The variogram sill. | float | | `fitnugget| Only valid if fit_method='manual'. The variogram nugget. | float | |fitsigma| Use a distance dependent weight on fits to favor closer bins. Do not set for | enum | |samplesize| Number of data points to use for the empirical variogram. This is used as a maximum sample size. If the passed data is larger, a random subsample will be taken from the supplied sample data. If instead a field is used, the field will be sub-sampled on along the spatial dimensions. If the field has a temporal dimension, it will be aggregated. If you set the sample_size to a **nagative value** (-1), the entire input data will be used, which may result in long runtimes. | integer | |field| Input data as a field, like a NetCDF variable or a GeoTiff. If these files get too large, the tool will operate on a sample. You can also force the sample, by setting a positive sample size. | file (extension) | |sample` | Input data as a spatial sample, like a CSV or parquet file. The tool will check for spatial columns named ('x', 'y', 'lon', 'lat', 'longitude', 'latitude') and use the first non-spatial column as the value column. If there is more than one non-spatial column, a cross-variogram is calculated. | file (extension) |

Kriging

| Parameter Name | Description | Data Type | |---|---|---| | grid | The grid size needs to be defined by a string like NNxMM, where NN are the number of rows and MM the number of columns | string | | algorithm | | enum | | mean | Real Mean value of the field. Only needed for Simple Kringing. | float | | drift_functions | Predefined drift function. Only needed for Universal Kriging. | enum | | variogram | A JSON file containing the variogram parameters as returned by Variogram.describe()['params']. You can use the 'variogram' tool to generate such a file in the output. You can only use the | file (extension) |

Simulation

| Parameter Name | Description | Data Type | |---|---|---| | grid | The grid size needs to be defined by a string like NNxMM, where NN are the number of rows and MM the number of columns | string | | n_simulations | Number of simulations to run. Defaults to 100 | integer | | seed | Seed for the random number generator. Defaults to 42. It is highly recommended to change this number on every run | integer | | variogram | A JSON file containing the variogram parameters as returned by Variogram.describe()['params']. You can use the 'variogram' tool to generate such a file in the output. You can only use the | file (extension) |

Why does this container look so weird?

This tool uses the Tool-Specs to standardize the input and output of the tool. It is based on the Python template.

The internal file structure is as follows:

/ |- in/ | |- inputs.json |- out/ | |- ... |- src/ | |- tool.yml | |- run.py

• inputs.json are parameters. Whichever framework runs the container, this is how parameters are passed.
• tool.yml is the tool specification. It contains metadata about the scope of the tool, the number of endpoints (functions) and their parameters
• run.py is the tool itself, or a Python script that handles the execution. It has to capture all outputs and either print them to console or create files in /out

How to use the image?

You pull this image from Github:

docker pull ghcr.io/vforwater/tbr_skgstat

Alternatively, you can build the image from within the root of this repo by docker build -t tbr_skgstat .

Use any tag you like. If you want to run and manage the container with toolbox-runner they should be prefixed by tbr_ to be recognized.

Alternatively, you can use GoRun (link and info will follow).

How to run?

If you do not use a package like GoRun or toolbox-runner to manage the docker images, you need to populate the /in/inputs.json with the parameters for the tool, copy the input data into the /in mount point and reference the data in the /in/inputs.json.

To invoke the docker container directly run something similar to: docker run --rm -it -v /path/to/local/in:/in -v /path/to/local/out:/out -e TOOL_RUN=variogram tbr_skgstat

Then, the output will be in your local out and based on your local input folder. Stdout and Stderr are also connected to the host. A Gorun based example will follow in a future release.