PYGEF

Simple parser for *.gef files. These are ASCII based files used for soil properties measurements. Compatible with Python 3.9.

Recently added the parsing of xml boreholes file, the xml parsing is still in a preliminary phase, not all the files are supported. If you find a file that doesn't work with pygef, please make an issue about it or PR :)

Installation

Latest stable version:

sh pip install pygef

Cutting-edge version (might break):

sh pip install git+https://github.com/cemsbv/pygef.git

CPT files

```python

from pygef import readcpt # read gef and xml files cptdata = readcpt("./my-cpt.xml") cptdata CPTData: {'broid': 'CPT000000099543', 'conediameter': 44, 'conesurfacearea': 1500, 'conesurfacequotient': 0.67, 'conetofrictionsleevedistance': 100, 'conetofrictionsleevesurfacearea': 22530, 'conetofrictionsleevesurfacequotient': 1.0, ... 'zlmporepressureu3after': None, 'zlmporepressureu3before': None} # access the underlying polars DataFrame under the data attribute cpt_data.data.head() shape: (5, 9) ┌────────────┬───────┬───────────┬────────────┬─────┬────────────┬────────────┬────────────┬────────────┐ │ penetratio ┆ depth ┆ elapsedTi ┆ coneResist ┆ ... ┆ inclinatio ┆ inclinatio ┆ localFrict ┆ frictionRa │ │ nLength ┆ --- ┆ me ┆ ance ┆ ┆ nNS ┆ nResultant ┆ ion ┆ tio │ │ --- ┆ f64 ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │ │ f64 ┆ ┆ f64 ┆ f64 ┆ ┆ i64 ┆ i64 ┆ f64 ┆ f64 │ ╞════════════╪═══════╪═══════════╪════════════╪═════╪════════════╪════════════╪════════════╪════════════╡ │ 0.0 ┆ 0.0 ┆ -9.99999 e ┆ -9.99999e5 ┆ ... ┆ -999999 ┆ -999999 ┆ -9.99999e5 ┆ -9.99999e5 │ │ ┆ ┆ 5 ┆ ┆ ┆ ┆ ┆ ┆ │ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤ │ 0.02 ┆ 0.02 ┆ 11.0 ┆ 2.708 ┆ ... ┆ 0 ┆ 0 ┆ 0.03 ┆ 0.6 │ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤ │ 0.04 ┆ 0.039 ┆ 13.0 ┆ 4.29 ┆ ... ┆ 0 ┆ 0 ┆ 0.039 ┆ 0.8 │ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤ │ 0.06 ┆ 0.059 ┆ 15.0 ┆ 5.124 ┆ ... ┆ 0 ┆ 0 ┆ 0.045 ┆ 0.9 │ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┤ │ 0.08 ┆ 0.079 ┆ 17.0 ┆ 5.45 ┆ ... ┆ 0 ┆ 0 ┆ 0.049 ┆ 1.0 │ └────────────┴───────┴───────────┴────────────┴─────┴────────────┴────────────┴────────────┴────────────┘ ```

Bore files

```python

from pygef import readbore # read gef and xml files boredata = readbore("./my-bore.xml") boredata BoreData: {'boreholecompleted': True, 'borerockreached': False, 'data': (13, 8), 'deliveredlocation': Location(srsname='urn:ogc:def:crs:EPSG::28992', x=158322.139, y=444864.706), 'deliveredverticalpositiondatum': 'nap', 'deliveredverticalpositionoffset': 10.773, 'deliveredverticalpositionreferencepoint': 'maaiveld', 'descriptionprocedure': 'ISO14688d1v2019c2020', 'finalboredepth': 12.0, 'finalsampledepth': 12.0, 'researchreportdate': datetime.date(2021, 10, 19)} # access the underlying polars DataFrame under the data attribute boredata.data.head() shape: (5, 8) ┌────────────┬────────────┬────────────┬──────────┬────────────┬────────────┬────────────┬─────────┐ │ upperboun ┆ lowerboun ┆ geotechnic ┆ color ┆ dispersed_ ┆ organicma ┆ sandmedia ┆ soildi │ │ dary ┆ dary ┆ alsoilna ┆ --- ┆ inhomogeni ┆ tterconte ┆ nclass ┆ st │ │ --- ┆ --- ┆ me ┆ str ┆ ty ┆ ntclass ┆ --- ┆ --- │ │ f64 ┆ f64 ┆ --- ┆ ┆ --- ┆ --- ┆ str ┆ list[f6 │ │ ┆ ┆ str ┆ ┆ bool ┆ str ┆ ┆ 4] │ ╞════════════╪════════════╪════════════╪══════════╪════════════╪════════════╪════════════╪═════════╡ │ 0.0 ┆ 1.0 ┆ zwakGrindi ┆ donkergr ┆ false ┆ nietOrgani ┆ middelgrof ┆ [0.2, │ │ ┆ ┆ gZand ┆ ijs ┆ ┆ sch ┆ 420 tot630u ┆ 0.0, │ │ ┆ ┆ ┆ ┆ ┆ ┆ m ┆ ... │ │ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 0.0] │ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ 1.0 ┆ 1.1 ┆ zwakGrindi ┆ donkergr ┆ false ┆ nietOrgani ┆ middelgrof ┆ [0.2, │ │ ┆ ┆ gZand ┆ ijs ┆ ┆ sch ┆ 420 tot630u ┆ 0.0, │ │ ┆ ┆ ┆ ┆ ┆ ┆ m ┆ ... │ │ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 0.0] │ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ 1.1 ┆ 2.0 ┆ zwakZandig ┆ standaar ┆ false ┆ nietOrgani ┆ null ┆ [0.0, │ │ ┆ ┆ eKleiMetGr ┆ dBruin ┆ ┆ sch ┆ ┆ 0.1, │ │ ┆ ┆ ind ┆ ┆ ┆ ┆ ┆ ... │ │ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 0.0] │ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ 2.0 ┆ 3.0 ┆ zwakZandig ┆ standaar ┆ false ┆ nietOrgani ┆ null ┆ [0.0, │ │ ┆ ┆ eKlei ┆ dGrijs ┆ ┆ sch ┆ ┆ 0.0, │ │ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ... │ │ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 0.0] │ ├╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ 3.0 ┆ 4.0 ┆ zwakZandig ┆ donkergr ┆ false ┆ nietOrgani ┆ null ┆ [0.0, │ │ ┆ ┆ eKlei ┆ ijs ┆ ┆ sch ┆ ┆ 0.0, │ │ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ... │ │ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 0.0] │ └────────────┴────────────┴────────────┴──────────┴────────────┴────────────┴────────────┴─────────┘ ```

Plotting

To use the plotting functionality pygef needs to be installed with the [plot] feature.

sh pip install pygef[plot]

```python from pygef import readcpt, readbore from pygef.plotting import plotcpt, plotbore

plot cpt file

plotcpt(readcpt("myfile.xml"))

plot a bore file

plotbore(readbore("myfile.xml")) ```

Documentation

Build the docs:

```bash python -m pip install --upgrade pip setuptools pip install -r requirements.txt pip install .

sphinx-build -b html docs public ```

Format

We format our code with black and isort.

bash black --config "pyproject.toml" . isort --settings-path "pyproject.toml" .

Lint

To maintain code quality we use the GitHub super-linter.

To run the linters locally, run the following bash script from the root directory:

bash docker run \ --env VALIDATE_ALL_CODEBASE=true \ --env DEFAULT_BRANCH=master \ --env RUN_LOCAL=true \ --env FIX_PYTHON_ISORT=true \ --env FIX_PYTHON_BLACK=true \ --env FIX_YAML_PRETTIER=true \ --env VALIDATE_JSCPD=false \ --env VALIDATE_CSS=false \ --env VALIDATE_BASH=false \ --env VALIDATE_YAML=false \ --env VALIDATE_PYTHON_PYLINT=false \ --env VALIDATE_NATURAL_LANGUAGE=false \ --env VALIDATE_MARKDOWN=false \ --env VALIDATE_MARKDOWN_PRETTIER=false \ --env LINTER_RULES_PATH=/ \ --env PYTHON_BLACK_CONFIG_FILE=pyproject.toml \ --env PYTHON_ISORT_CONFIG_FILE=pyproject.toml \ --env PYTHON_MYPY_CONFIG_FILE=pyproject.toml \ --env PYTHON_FLAKE8_CONFIG_FILE=.flake8 \ -v $(pwd):/tmp/lint ghcr.io/super-linter/super-linter:v8

UnitTest

Test the software with the use of coverage:

```bash python -m pip install --upgrade pip setuptools pip install -r requirements.txt pip install -e .[test] .[map] .[plot]

coverage run -m pytest ```

Requirements

Requirements are autogenerated by pip-compile with python 3.9

bash pip-compile --extra=plot --extra=test --extra=docs --extra=lint --extra=map --output-file=requirements.txt pyproject.toml

To update the requirements within the defined ranges, run:

bash pip-compile --upgrade --extra=plot --extra=test --extra=docs --extra=lint --extra=map --output-file=requirements.txt pyproject.toml