Rarr

A simple native R reader for Zarr Arrays

https://github.com/Huber-group-EMBL/Rarr

Keywords

Keywords from Contributors

Repository

A simple native R reader for Zarr Arrays

README.Rmd

---
title: "Zarr arrays with Rarr"
author: "Mike L. Smith"
output:
  github_document:
    toc: true
    toc_depth: 2
---


| GitHub Actions | Bioconductor Build Sysytem | Test Coverage |
|:--------------:|:-------------:|:-----:|
| [![Package Checks](https://github.com/grimbough/Rarr/actions/workflows/main.yml/badge.svg)](https://github.com/grimbough/Rarr/actions/workflows/main.yml) | [![Bioconductor Status](https://bioconductor.org/shields/build/devel/bioc/Rarr.svg)](https://bioconductor.org/checkResults/devel/bioc-LATEST/Rarr/) | [![Codecov test coverage](https://codecov.io/gh/grimbough/Rarr/branch/devel/graph/badge.svg)](https://app.codecov.io/gh/grimbough/Rarr?branch=devel) |


```{r, config, echo = FALSE}
knitr::opts_chunk$set(fig.path = "inst/rmd/imgs/", dev = "jpeg")
```

# Introduction to Rarr

The Zarr specification defines a format for chunked, compressed, N-dimensional
arrays.  It's design allows efficient access to subsets of the stored array, and
supports both local and cloud storage systems. Zarr is experiencing increasing
adoption in a number of scientific fields, where multi-dimensional data are
prevalent.

**Rarr** is intended to be a simple interface to reading and writing individual
Zarr arrays. It is developed
in R and C with no reliance on external libraries or APIs for interfacing with
the Zarr arrays. Additional compression libraries (e.g. blosc) are bundled with
**Rarr** to provide support for datasets compressed using these tools.

## Limitations with **Rarr**

**Rarr currently only works with [Zarr specification version 2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html).
Support for [version 3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html) is actively being worked on.**

If you know about Zarr arrays already, you'll probably be aware they can be
stored in hierarchical groups, where additional meta data can explain the
relationship between the arrays.  Currently, **Rarr** is not designed to be
aware of these hierarchical Zarr array collections. However, the component
arrays can be read individually by providing the path to them directly.

Currently, there are also limitations on the Zarr datatypes that can be accessed
using **Rarr**.  For now most numeric types can be read into R, although in some
instances e.g. 64-bit integers there is potential for loss of information.
Writing is more limited with support only for datatypes that are supported
natively in R and only using the column-first representation.

# Quick start guide

```{r, child=c('inst/rmd/quick_start.Rmd')}
```


# Current Status

## Reading and Writing

Reading Zarr arrays is reasonably well supported. Writing is available, but is more limited.  Both aspects are under active development.

### Data Types

Currently there is only support for reading and writing a subset of the possible datatypes
that can be found in a Zarr array.  In some instances there are also limitations on the
datatypes natively supported by R, requiring conversion from the Zarr datatype.  The table below summarises the current status of
datatype support.  It will be updated as progress is made.

| Zarr Data Type | Status
(reading / writing) | Notes |
|-----------|:--------------:|-------|
|`boolean`  | ✔ / ❌ | |
|`int8`     | ✔ / ❌ | |
|`uint8` | ✔ / ❌ | |
|`int16` | ✔ / ❌ | |
|`uint16`| ✔ / ❌ | |
|`int32` | ✔ / ✔ | |
|`uint32`| ✔ / ❌ | Values outside the range of `int32` are converted to `NA`.  Future plan is to allow conversion to `double` or use the [bit64](https://cran.r-project.org/package=bit64) package. |
|`int64` | ✔ / ❌ | Values outside the range of `int32` are converted to `NA`. Future plan is to allow conversion to `double` or use the [bit64](https://cran.r-project.org/package=bit64) package. |
|`uint64`| ✔ / ❌ | Values outside the range of `int32` are converted to `NA`. Future plan is to allow conversion to `double` or use the [bit64](https://cran.r-project.org/package=bit64) package. |
|`half` / `float16`   | ✔ / ❌ | Converted to `double` in R.  No effort is made to assess loss of precision due to conversion.  |
|`single` / `float32` | ✔ / ❌ | Converted to `double` in R.  No effort is made to assess loss of precision due to conversion. |
|`double` / `float64` | ✔ / ✔ | |
|`complex`            | ❌ / ❌ | |
|`timedelta`          | ❌ / ❌ | |
|`datetime`           | ❌ / ❌ | |
|`string`             | ✔ / ✔ | |
|`Unicode`            | ✔ / ✔ | |
|`void *`             | ❌ / ❌ | |
| Structured data types | ❌ / ❌ | |

### Compression Tools

| Data Type   | Status
(reading / writing) | Notes |
|-------------|:-------------------:|-------|
|`zlib / gzip`| ✔ / ✔ | Only system default compression level (normally 6) is enabled for writing. |
|`bzip2`      | ✔ / ✔ | Only compression level 9 is enabled for writing. |
|`blosc`      | ✔ / ✔ | Only `lz4` compression level 5 is enabled for writing. |
|`LZMA `      | ✔ / ✔ | |
|`LZ4`        | ✔ / ✔ | |
|`Zstd`       | ✔ / ✔ | |

Please open an [issue](https://github.com/grimbough/Rarr/issues) if support for a required compression tool is missing.

### Filters

The is currently no support for additional filters.  Please open an [issue](https://github.com/grimbough/Rarr/issues) if you require filter support.

# Required system libraries

To provide support for BLOSC and zstd compression tools Rarr links against libraries providing these tools.  If you have them installed on your system Rarr will attempt to use those versions.  If they are not detected then Rarr will compile and use versions that are distributed with the package.  Either way the functionality will available, however if you are using the system libraries and then later remove them Rarr may fail to work correctly.

This only concerns users installing the package from source. If you are using the pre-built binaries for Windows or Mac OSX distributed by Bioconductor then this should not be an issue for you.

Owner

GitHub Events

Committers

Last synced: 8 months ago

Issues and Pull Requests

Last synced: 6 months ago