lazysf

Delayed Read for GDAL Vector Data Sources

https://github.com/hypertidy/lazysf

Keywords

Repository

Delayed Read for GDAL Vector Data Sources

README.Rmd

---
output: github_document
---



```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
library(dplyr)
library(sf)
```

# lazysf


[![CRAN status](https://www.r-pkg.org/badges/version/lazysf)](https://CRAN.R-project.org/package=lazysf)
[![](http://cranlogs.r-pkg.org/badges/last-month/lazysf?color=green)](https://cran.r-project.org/package=lazysf)


The goal of lazysf is to provide interactive delayed read of GDAL vector data sources. 

Note: very soon a release of [gdalraster](https://CRAN.R-project.org/package=gdalraster) will supersede what lazysf can
do and we recommend using that instead. 

Vector data sources, drawings (a.k.a. "shapefiles") are files or web services or databases that provide tables of data fields. These fields
may include spatial geometry data such as points, lines, polygons, and other planar types composed of paths of coordinates. 

lazysf uses the dplyr/dbplyr 'tbl_lazy' mechanism by providing a GDAL DBI-backend like many database packages in R.  The convenience function `lazysf()` provides a single-argument wrapper around the database-like workflows. 

See it in action!

```{r sf-action}
library(lazysf)
library(sf)
library(dplyr)

url <- "https://github.com/Nowosad/spData/raw/master/inst/shapes/NY8_bna_utm18.gpkg"
(x <- lazysf(url))
x %>% distinct(AREANAME) %>% arrange(AREANAME) 

plot(st_as_sf(x %>% 
                dplyr::filter(!(AREANAME %LIKE% "Ca%" | AREANAME %LIKE% "Bi%")) %>% 
                dplyr::select(AREANAME, geom)))
```

## Limitations

This is very largely format dependent, and by "format" we mean the *driver* as provided by GDAL.   

We make no claims about performance or convenience, it will be affected by your system and your sf installation - lazysf just takes you closer the GDAL capabilities. 

Performance can be excellent, and may be very competitive compared to reading an entire data source layer into memory. Really good drivers include ESRI Shapefile, Geopackage, PostgreSQL/PostGIS, MapInfo File, ESRI FileGDB, but there are dozens to choose from. 

A query on a CSV, GeoJSON, or KML file (local or remote) is entirely subject to the performance of the matching [GDAL driver](https://gdal.org/en/stable/drivers/vector/index.html).

* big text files will be slow, they aren't a format suitable for database-like access
* geometry is not automatic for real database formats, and will depend on the SQL used
* geometry *is* automatic for non-DB formats
* non-DB formats without a geometry column name will be called `_ogr_geometry_`, other non-DB formats like
ESRI's geodatabase have other names like `SHAPE`
* non-DB formats have access to special variable names and functions, listed on the [OGRSQL page](https://gdal.org/en/stable/user/ogr_sql_dialect.html). 

Real DBs don't have these special OGRSQL features, but they do have their own special syntax which for the most part can be sent straight through. 

When using dplyr verbs (`filter()`, `select()`, `mutate()`, `transmute()`,
`arrange()`, `left_join()`, ...) we are also subject to the rules of SQL
translation. There are no specific ones provided by lazysf but that might
change.

Wrappers around lazysf could provide more specific tools for particular formats. 

### Can't we just do this with sf's query argument?

Yes (actually that is what lazysf uses) but with sf alone you get a fully
materialized sf data frame, so you better get that query right first time!

With lazysf you get some control over intermediate steps, potentially expensive
queries will only be run for a preview of the data until you are ready to fetch
it.

## Installation

You can install the dev version of lazysf from [GitHub](https://github.com/hypertidy/lazysf) with:

``` r
# Enable this universe
options(repos = c(
    hypertidy = 'https://hypertidy.r-universe.dev',
    CRAN = 'https://cloud.r-project.org'))

# Install some packages
install.packages('lazysf')
```

## Example

This is a basic example. 

```{R basic}
library(lazysf)
f <- system.file("gpkg/nc.gpkg", package = "sf", mustWork = TRUE)

## specify only the data source
lazysf(f)

## specify the data source and a query to run
lazysf(f, query = "SELECT AREA, FIPS, geom FROM \"nc.gpkg\" WHERE AREA < 0.1")

## specify the data source and the table/layer to access
lazysf(f, layer = "nc.gpkg") %>% 
  dplyr::select(AREA, FIPS, geom) %>% 
  dplyr::filter(AREA < 0.1)


## above was a real database (Geopackage), now with an actual shapefile
shp <- lazysf(system.file("shape/nc.shp", package = "sf", mustWork = TRUE))
library(dplyr)
shp %>%
 filter(NAME %LIKE% 'A%') %>%
 mutate(abc = 1.3) %>%
 select(abc, NAME, `_ogr_geometry_`) %>%
 arrange(desc(NAME))  #%>% show_query()
```

Online sources can also work if your build of sf supports. 

```{r online}
# online sources can work
geojson <- file.path("https://raw.githubusercontent.com/SymbolixAU",
                      "geojsonsf/master/inst/examples/geo_melbourne.geojson")
lazysf(geojson)
```


Also works on PostgreSQL and many others as per GDAL vector driver support.   


To create a connection string for [GDAL for PostgreSQL](https://gdal.org/en/stable/drivers/vector/pg.html) use something like

```R
DSN <- glue::glue("PG:host='{host}' dbname='{dbname}' user='{user}' password='{password}'")

dbConnect(SFSQL(), DSN)
```

but the same can be done with generic DBI and (for example) the [Rpostgres package](https://rpostgres.r-dbi.org/). With `SFSQL()` we just know that it's executed by GDAL (via sf). 


## Some drivers are related

Note that GDAL drivers can be confusing, and it can be important to see the behaviours GDAL will provide by default. Here see that we can read from a Geopackage file but not as it was intended. We
have used the driver-prefix to make GDAL choose its SQLite driver rather than the Geopackage driver. 

```{r Geopackage-SQLite}
library(lazysf)
gpkgfile <- system.file("gpkg/nc.gpkg", package = "sf", mustWork = TRUE)
lazysf(glue::glue("SQLite:{gpkgfile}"))
```

That's not very spatial, but we can dig in to find out what else is there. 



  
---
## Code of Conduct

Please note that the lazysf project is released with a [Contributor Code of
Conduct](https://contributor-covenant.org/version/2/0/CODE_OF_CONDUCT.html). By
contributing to this project, you agree to abide by its terms.

Owner

GitHub Events

Committers

Last synced: about 1 year ago

Issues and Pull Requests

Last synced: 11 months ago

Packages