Data

Tools

Indexes

Applications

Experiments

Open Source Science

io.jhdf:jhdf

A pure Java HDF5 library

https://github.com/jamesmudd/jhdf

Science Score: 77.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 3 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: zenodo.org
  • ✓
    Committers with academic emails
    2 of 15 committers (13.3%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (9.3%) to scientific vocabulary

Keywords

bigdata file-format hdf5 java

Keywords from Contributors

interactive bioinformatics genomics earth-observation mesh interpretability profiles sequences generic projection
Last synced: 4 months ago · JSON representation ·

Repository

A pure Java HDF5 library

Basic Info
  • Host: GitHub
  • Owner: jamesmudd
  • License: mit
  • Language: Java
  • Default Branch: master
  • Homepage: http://jhdf.io
  • Size: 4.45 MB
Statistics
  • Stars: 161
  • Watchers: 11
  • Forks: 41
  • Open Issues: 23
  • Releases: 49
Topics
bigdata file-format hdf5 java
Created about 7 years ago · Last pushed 4 months ago
Metadata Files
Readme Changelog Funding License Code of conduct Citation Security

README.md

jHDF - Pure Java HDF5 library

jHDF CI Coverage Maven Central Javadocs APIdia JetBrains Supported DOI

This project is a pure Java implementation for accessing HDF5 files. It is written from the file format specification and is not using any HDF Group code, it is not a wrapper around the C libraries. The file format specification is available from the HDF Group here. More information on the format is available on Wikipedia. I presented a webinar about jHDF for the HDF Group which is available on YouTube the example code used and slides can be found here.

The intention is to make a clean Java API to access HDF5 data. Currently, reading is very well-supported and writing supports limited use cases. For progress see the change log. Java 17 and 21 are officially supported.

Here is an example of reading a dataset with jHDF (see ReadDataset.java)

java try (HdfFile hdfFile = new HdfFile(Paths.get("/path/to/file.hdf5"))) { Dataset dataset = hdfFile.getDatasetByPath("/path/to/dataset"); // data will be a Java array with the dimensions of the HDF5 dataset Object data = dataset.getData(); }

For an example of traversing the tree inside a HDF5 file see PrintTree.java.

An example of writing a file jhdf.hdf5 containing a group group, with two datasets ints and doubles

java try (WritableHdfFile hdfFile = HdfFile.write(Paths.get("jhdf.hdf5"))) { WritableGroup group = hdfFile.putGroup("group"); group.putDataset("ints", new int[] {1, 2, 3, 4}); group.putDataset("doubles", new double[] {1.0, 2.0, 3.0, 4.0}); } See WriteHdf5.java for a more extensive complete example. Note: writing files is still an early feature with many more functions to be added.

For more examples see package io.jhdf.examples

Why should I use jHDF?

  • Easy integration with JVM based projects. The library is available on Maven Central, GitHub Packages, and JitPack.io so using it should be as easy as adding any other dependency. To use the libraries supplied by the HDF Group you need to load native code, which means you need to handle this in your build, and it complicates distribution of your software on multiple platforms.
  • The API design intends to be familiar to Java programmers, so hopefully it works as you might expect. (If this is not the case, open an issue with suggestions for improvement)
  • No use of JNI, so you avoid all the issues associated with calling native code from the JVM.
  • Fully debug-able you can step fully through the library with a Java debugger.
  • Provides access to datasets ByteBuffers to allow for custom reading logic, or integration with other libraries.
  • Allows integration with different file system implementations (e.g. zip files, or in-memory) and the ability to read remote files from web servers or object storage like S3.
  • Integration with Java logging via SLF4J
  • Performance? Maybe, the library uses Java NIO MappedByteBuffers which should provide fast file access. In addition, when accessing chunked datasets, the library is parallelized to take advantage of modern CPUs. jHDF will also allow parallel reading of multiple datasets or multiple files. I have seen cases where jHDF is significantly faster than the C libraries, but as with all performance issues, it is case-specific, so you will need to do your own tests on the cases you care about. If you do run tests, please post the results so everyone can benefit, here are some results I am aware of:
  • Security - jHDF is pure Java, and therefore benefits from the memory safety provided by the JVM. The HDF5 Group library is written using non-memory safe languages, therefore susceptible to memory related security bugs.

Why should I not use jHDF?

If you use HDF5 from Java I think you should give it a try. However, if jHDF does not yet support a feature you need, you should receive a UnsupportedHdfException, open an issue and support can be added. For scheduling, the features which will allow the most files to be read/written are prioritized. If you really want to use a new feature, feel free to work on it and open a PR. Any help is much appreciated.

Why did I start jHDF?

Mostly it's a challenge, HDF5 is a fairly complex file format with lots of flexibility, writing a library to access it is interesting. Also, as a widely used file format for storing scientific, engineering, and commercial data, it would seem like a good idea to be able to access HDF5 files with more than one library. In particular, JVM languages are among the most widely used, so having a native HDF5 implementation seems useful.

Developing jHDF

  • Fork this repository and clone your fork
  • Inside the jhdf directory run ./gradlew build (./gradlew.bat build on Windows) this will run the build and tests fetching dependencies.
  • Import the Gradle project jhdf into your IDE.
  • Make your changes and add tests.
  • Run ./gradlew check to run the build and tests.
  • Once you have made any changes, please open a pull request.

To see other available Gradle tasks run ./gradlew tasks

If you have read this far, please consider staring at this repo. If you are using jHDF in a commercial product, please consider making a donation. Thanks!

Owner

  • Name: James Mudd
  • Login: jamesmudd
  • Kind: user
  • Location: Shropshire, UK
  • Company: @OspreyCharging

Scientist turned programmer. Now a full stack developer working in Kotlin, Java, Typescript and cloud.

Citation (CITATION.cff) 

cff-version: 1.2.0
title: "jHDF"
message: "If you use this software, please cite it using the metadata from this file."
type: "software"
doi: "10.5281/zenodo.3996097"
url: "http://jhdf.io/"
authors:
  - given-names: "James"
    family-names: "Mudd"
    email: "james.mudd@gmail.com"
    orcid: "https://orcid.org/0000-0002-4382-8903"

GitHub Events

Total
  • Create event: 86
  • Release event: 9
  • Issues event: 24
  • Watch event: 23
  • Delete event: 77
  • Issue comment event: 136
  • Push event: 116
  • Pull request review comment event: 27
  • Pull request review event: 24
  • Pull request event: 161
  • Fork event: 5
Last Year
  • Create event: 86
  • Release event: 9
  • Issues event: 24
  • Watch event: 23
  • Delete event: 77
  • Issue comment event: 136
  • Push event: 116
  • Pull request review comment event: 27
  • Pull request review event: 24
  • Pull request event: 161
  • Fork event: 5

Committers

Last synced: 7 months ago

All Time
  • Total Commits: 1,491
  • Total Committers: 15
  • Avg Commits per committer: 99.4
  • Development Distribution Score (DDS): 0.209
Past Year
  • Commits: 248
  • Committers: 8
  • Avg Commits per committer: 31.0
  • Development Distribution Score (DDS): 0.403
Top Committers
Name Email Commits
James Mudd j****d@g****m 1,179
dependabot[bot] 4****] 224
Thomas Reimonn t****n@u****u 23
Tobias Brunsch T****h@k****m 15
Tobias Brunsch t****h@a****e 14
Jonathan Shook j****k@g****m 10
Stefan Obermeier s****r@s****e 8
alex a****n@c****m 4
Raedel, Martin m****l@d****e 3
Matthias Mailänder m****r@l****m 3
jczogalla j****a@r****m 2
marcobitplane 1****e 2
Gisa Meier g****r@r****m 2
Brad Hards b****h@f****t 1
Ivan Wick i****k@b****m 1
Committer Domains (Top 20 + Academic)
rapidminer.com: 2 bionano.com: 1 frogmouth.net: 1 lablicate.com: 1 dlr.de: 1 corista.com: 1 seeburger.de: 1 aol.de: 1 kistler.com: 1 umassmed.edu: 1

Issues and Pull Requests

Last synced: 4 months ago

All Time
  • Total issues: 37
  • Total pull requests: 455
  • Average time to close issues: 11 months
  • Average time to close pull requests: 11 days
  • Total issue authors: 19
  • Total pull request authors: 10
  • Average comments per issue: 3.03
  • Average comments per pull request: 0.87
  • Merged pull requests: 351
  • Bot issues: 0
  • Bot pull requests: 291
Past Year
  • Issues: 13
  • Pull requests: 176
  • Average time to close issues: 9 days
  • Average time to close pull requests: 4 days
  • Issue authors: 10
  • Pull request authors: 9
  • Average comments per issue: 1.0
  • Average comments per pull request: 0.75
  • Merged pull requests: 145
  • Bot issues: 0
  • Bot pull requests: 91
View more stats
Top Authors
Issue Authors
  • jamesmudd (14)
  • Apollo3zehn (2)
  • calum-maclean-ul (2)
  • Heiney-Upenn (2)
  • tianxie (2)
  • tischi (2)
  • sfc-gh-mrojas (1)
  • raedma (1)
  • CodeByDrescher (1)
  • mullerhai (1)
  • holderl5 (1)
  • zhuam (1)
  • tbrunsch (1)
  • jshook (1)
  • ebremer (1)
Pull Request Authors
  • dependabot[bot] (291)
  • jamesmudd (144)
  • jshook (4)
  • thomas-reimonn (4)
  • Mailaender (3)
  • marcobitplane (2)
  • ivanwick (2)
  • cstoeckl (2)
  • tbrunsch (2)
  • bradh (1)
Top Labels
Issue Labels
enhancement (15) bug (7)
Pull Request Labels
dependencies (292)

Packages

  • Total packages: 2
  • Total downloads: unknown
  • Total docker downloads: 612,356
  • Total dependent packages: 6
    (may contain duplicates)
  • Total dependent repositories: 11
    (may contain duplicates)
  • Total versions: 110
proxy.golang.org: github.com/jamesmudd/jhdf
  • Versions: 57
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 5.3%
Average: 5.5%
Dependent repos count: 5.7%
Last synced: 4 months ago
repo1.maven.org: io.jhdf:jhdf

A pure Java HDF5 library

  • Versions: 53
  • Dependent Packages: 6
  • Dependent Repositories: 11
  • Docker Downloads: 612,356
Rankings
Docker downloads count: 2.1%
Dependent repos count: 7.4%
Dependent packages count: 9.7%
Average: 15.4%
Stargazers count: 27.5%
Forks count: 30.1%
Last synced: 4 months ago

Dependencies

.github/workflows/auto-dependency-updates.yml actions
  • dependabot/fetch-metadata v1.3.5 composite
.github/workflows/ci.yml actions
  • actions/checkout v3 composite
  • actions/setup-java v3 composite
.github/workflows/codeql-analysis.yml actions
  • actions/checkout v3 composite
  • github/codeql-action/analyze v2 composite
  • github/codeql-action/autobuild v2 composite
  • github/codeql-action/init v2 composite
.github/workflows/release.yml actions
  • actions/checkout v3 composite
  • actions/setup-java v3 composite
  • actions/upload-artifact v3 composite
.github/pom.xml maven
  • com.ning:compress-lzf 1.1
  • org.apache.commons:commons-lang3 3.12.0
  • org.slf4j:slf4j-api 1.7.30
jhdf/build.gradle maven
  • org.xerial.snappy:snappy-java 1.1.8.4 testImplementation
.github/workflows/dependency-review.yml actions
  • actions/checkout v3 composite
  • actions/dependency-review-action v3 composite
.github/workflows/qodana.yml actions
  • JetBrains/qodana-action v2022.3.4 composite
  • actions/checkout v3 composite
.github/workflows/sonarqube.yml actions
  • actions/checkout v4 composite
  • actions/setup-java v4 composite