Science Score: 64.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: ncbi.nlm.nih.gov -
✓Committers with academic emails
1 of 38 committers (2.6%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (14.1%) to scientific vocabulary
Keywords
Keywords from Contributors
Repository
Cloud-native, service based access to HDF data
Basic Info
- Host: GitHub
- Owner: HDFGroup
- License: apache-2.0
- Language: Python
- Default Branch: master
- Homepage: https://www.hdfgroup.org/solutions/hdf-kita/
- Size: 8.17 MB
Statistics
- Stars: 146
- Watchers: 21
- Forks: 58
- Open Issues: 33
- Releases: 15
Topics
Metadata Files
README.md
HSDS (Highly Scalable Data Service) - REST-based service for HDF5 data
Introduction
HSDS is a web service that implements a REST-based web service for HDF5 data stores. Data can be stored in either a POSIX files system, or using object-based storage such as AWS S3, Azure Blob Storage, or MinIO. HSDS can be run a single machine with or without Docker or on a cluster using Kubernetes (or AKS on Microsoft Azure).
Quick Start
With Github codespaces
Launch a Codespaces environment by clicking the banner "Open in GitHub Codespaces". Once the codespace is ready, type:
python testall.py in the terminal window to run the test suite.
On your desktop/laptop
Make sure you have Python 3 and Pip installed, then:
- Run install:
$ ./build.sh --no-lint --no-dockerfrom source tree OR install from pypi:$ pip install hsds - Create a directory the server will use to store data, example:
$ mkdir ~/hsds_data - Start server:
$ hsds --root_dir ~/hsds_data - Run the test suite. In a separate terminal run:
- Set username: `$ export USERNAME=$USER`
- Set userpassword: `$ export USERPASSWORD=$USER`
- Set admin name:
$ export ADMIN_USERNAME=$USER - Set admin password:
$ export ADMIN_PASSWORD=$USER - Run test suite:
$ python testall.py --skip_unit
- (Optional) Install the h5pyd package for an h5py compatible api and tool suite: https://github.com/HDFGroup/h5pyd
- (Optional) Post install setup (test data, home folders, cli tools, etc): docs/post_install.md
To shut down the server, and the server is not running in Docker, just control-C.
If using docker, run: $ ./stopall.sh
Note: passwords can (and should for production use) be modified by changing values in hsds/admin/config/password.txt and rebuilding the docker image. Alternatively, an external identity provider such as Azure Active Directory or KeyCloak can be used. See: docs/azureadsetup.md for Azure AD setup instructions or docs/keycloak_setup.md for KeyCloak.
Detailed Install Instructions
On AWS
For complete instructions to install on a single Azure VM with Docker:
For complete instructions to install on AWS Kubernetes Service (EKS):
For complete instructions to install on AWS Lambda:
- See: docs/awslambdasetup.md.
On Azure
For complete instructions to install on a single Azure VM with Docker:
For complete instructions to install on Azure Kubernetes Service (AKS):
On Prem (POSIX-based storage)
For complete instructions to install on a desktop or local server:
On DCOS (BETA)
For complete instructions to install on DCOS:
General Install Topics
Setting up docker:
Post install setup and testing:
Authorization, ACLs, and Role Based Access Control (RBAC):
Writing Client Applications
As a REST service, clients be developed using almost any programming language. The test programs under: hsds/test/integ illustrate some of the methods for performing different operations using Python and HSDS REST API (using the requests package).
The related project: https://github.com/HDFGroup/h5pyd provides a (mostly) h5py-compatible interface to the server for Python clients.
For C/C++ clients, the HDF REST VOL is a HDF5 library plugin that enables the HDF5 API to read and write data using HSDS. See: https://github.com/HDFGroup/vol-rest. Note: requires v1.12.0 or greater version of the HDF5 library.
Uninstalling
HSDS only modifies the storage location that it is configured to use, so to uninstall just remove source files, Docker images, and S3 bucket/Azure Container/directory files.
Reporting bugs (and general feedback)
Create new issues at http://github.com/HDFGroup/hsds/issues for any problems you find.
For general questions/feedback, please use the HSDS forum: https://forum.hdfgroup.org/c/hsds.
License
HSDS is licensed under an APACHE 2.0 license. See LICENSE in this directory.
Azure Marketplace
VM Offer for Azure Marketplace. HSDS for Azure Marketplace provides an easy way to setup a Azure instance with HSDS. See: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/thehdfgroup1616725197741.hsdsazurevm?tab=Overview for more information.
Websites
- Main website: https://www.hdfgroup.org/solutions/highly-scalable-data-service-hsds/
- Source code: https://github.com/HDFGroup/hsds
- Forum: https://forum.hdfgroup.org/c/hsds
- Documentation: https://support.hdfgroup.org/documentation/index.html
- REST API: https://github.com/HDFGroup/hdf-rest-api
Other useful resources
HDF Group Blog Posts
- Web Caching: https://www.hdfgroup.org/2022/10/improve-hdf5-performance-using-caching/
- HSDS Streaming: https://www.hdfgroup.org/2022/08/hsds-streaming/
- Cloud Storage Options for HDF5: https://www.hdfgroup.org/2022/08/cloud-storage-options-for-hdf5/
- HSDS Docker Images: https://www.hdfgroup.org/2022/07/hsds-docker-images/
- HSDS Container Types: https://www.hdfgroup.org/2022/07/deep-dive-hsds-container-types/
- Using Multiprocessing in Python: https://www.hdfgroup.org/2022/06/speed-up-cloud-access-using-multiprocessing/
- Biosimulations - case study with HSDS and Vega: https://www.hdfgroup.org/2022/02/biosimulations-a-platform-for-sharing-and-reusing-biological-simulations/
- HSDS for Microsoft Azure: https://www.hdfgroup.org/2021/08/hsds-for-azure/
- New Features in HSDS v0.6: https://www.hdfgroup.org/2020/10/new-features-in-hsds-version-0-6/
- HSDS Security: https://hdfgroup.org/wp/2015/12/serve-protect-web-security-hdf5
- HDF for the Web: HDF Server: https://www.hdfgroup.org/2015/04/hdf5-for-the-web-hdf-server/
External Blogs and Articles
- A RESTful Meeting Between MATLAB and HDF Server: https://www.mathworks.com/matlabcentral/fileexchange/59072-a-restful-meeting-between-matlab-and-hdf-server-web-based-hdf5-access-using-matlab
- AWS Big Data Blog: https://aws.amazon.com/blogs/big-data/power-from-wind-open-data-on-aws/
Slide Decks
- HSDS v0.7 New Features, EUHUG 2022: https://www.hdfgroup.org/wp-content/uploads/2022/05/HSDS_New_Feautres_7.0.pdf
- HSDS Serverless, EUHUG 2021: https://www.hdfgroup.org/wp-content/uploads/2021/07/ServerlessHSDS.pdf
- HSDS REST, HUG 2020: https://www.hdfgroup.org/wp-content/uploads/2020/10/HSDS_Rest_Service_HDF5_Readey.pdf
- HSDS with Jupyter, ESIP 2018: https://www.slideshare.net/HDFEOS/hdf-kita-lab-jupyterlab-hdf-service
- HDF Data Services, SciPy17: http://s3.amazonaws.com/hdfgroup/docs/hdf_data_services_scipy2017.pdf
Videos
- HSDS Webinar: https://www.youtube.com/watch?v=9b5TO7drqqE
- HSDS Overview, Allotrope Connect Day: https://www.youtube.com/watch?v=nRHXEkhlfZ0
- The Use of HSDS on SlideRule, HUG 2020: https://www.youtube.com/watch?v=i-KIoGqdEMg
- HDF Data Services, SciPy 2017: https://www.youtube.com/watch?v=EmnCz1Hg-VM
- RESTful HDF, SciPy 2015: https://www.youtube.com/watch?v=JSFZ3i3WcjQ
Papers
- restfulSE: A semantically rich interface for cloud-scale genomics with Bioconductor: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6392152
- RESTful HDF5 White Paper: https://www.hdfgroup.org/pubs/papers/RESTful_HDF5.pdf
Owner
- Name: The HDF Group
- Login: HDFGroup
- Kind: organization
- Email: help@hdfgroup.org
- Location: Champaign, IL, USA
- Website: https://www.hdfgroup.org
- Twitter: hdf5
- Repositories: 78
- Profile: https://github.com/HDFGroup
Tools and technologies to support the Hierarchical Data Format (HDF)
Citation (CITATION.cff)
cff-version: 1.2.0
title: 'Highly Scalable Data Service (HSDS)'
message: >-
If you use this software, please cite it using the
metadata from this file.
type: software
authors:
- name: The HDF Group
website: 'https://www.hdfgroup.org'
repository-code: 'https://github.com/HDFGroup/hsds'
url: 'https://www.hdfgroup.org/solutions/highly-scalable-data-service-hsds/'
GitHub Events
Total
- Create event: 16
- Issues event: 10
- Release event: 2
- Watch event: 15
- Delete event: 14
- Issue comment event: 25
- Push event: 53
- Pull request review comment event: 7
- Pull request review event: 16
- Pull request event: 46
- Fork event: 5
Last Year
- Create event: 16
- Issues event: 10
- Release event: 2
- Watch event: 15
- Delete event: 14
- Issue comment event: 25
- Push event: 53
- Pull request review comment event: 7
- Pull request review event: 16
- Pull request event: 46
- Fork event: 5
Committers
Last synced: 9 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| jreadey | j****y@h****g | 1,572 |
| mattjala | 1****a | 58 |
| Andrew Michaelis | a****c@h****g | 39 |
| Thomas VINCENT | t****t@e****r | 27 |
| Bilal Shaikh | b****2@g****m | 24 |
| Aleksandar Jelenak | a****k@h****g | 19 |
| Jose Francisco Guerrero | j****o@t****m | 15 |
| dependabot[bot] | 4****] | 11 |
| s004pmg | s****g | 11 |
| Jose Francisco Guerrero | j****o | 10 |
| Kate Kader | k****r@m****m | 9 |
| loichuder | l****r@e****r | 9 |
| Jonathan Jara-Almonte | j****m@p****v | 7 |
| Daniel Felipe | p****2@g****m | 5 |
| Michael Bonfils | b****l@p****m | 5 |
| Jordan Henderson | j****n@h****g | 4 |
| Daniel Olshansky | o****l@g****m | 4 |
| Prashant Karbhari | p****b@m****m | 4 |
| Thiago M. de C. Marques | t****e@y****r | 4 |
| Mark Bruggemann | m****k@e****o | 3 |
| Jonathan Jara-Almonte | j****t@l****n | 3 |
| Akshay Thakare | a****a@g****m | 2 |
| H. Joe Lee | h****e@h****g | 2 |
| Lori Cooper | l****r@h****g | 2 |
| Paul Edwards | p****s@n****v | 2 |
| Josdin Daniel | j****2@g****m | 2 |
| EC2 Default User | e****r@i****l | 2 |
| trmt | t****1@m****u | 1 |
| jananzhu | j****u@g****m | 1 |
| Dana Robinson | d****s@h****g | 1 |
| and 8 more... | ||
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 95
- Total pull requests: 229
- Average time to close issues: 4 months
- Average time to close pull requests: 17 days
- Total issue authors: 32
- Total pull request authors: 16
- Average comments per issue: 3.46
- Average comments per pull request: 0.58
- Merged pull requests: 195
- Bot issues: 0
- Bot pull requests: 23
Past Year
- Issues: 7
- Pull requests: 50
- Average time to close issues: 6 days
- Average time to close pull requests: 2 days
- Issue authors: 6
- Pull request authors: 5
- Average comments per issue: 1.29
- Average comments per pull request: 0.3
- Merged pull requests: 40
- Bot issues: 0
- Bot pull requests: 6
Top Authors
Issue Authors
- jreadey (29)
- mattjala (22)
- bilalshaikh42 (6)
- assaron (3)
- ron-kuhn (2)
- jananzhu (2)
- AboPlus (2)
- jonatantreijs (2)
- TManhente (2)
- gheber (2)
- Will-Gorman (1)
- WowIDidntKnow (1)
- weiyuan-jiang (1)
- Leonard-Mueller (1)
- satelite2517 (1)
Pull Request Authors
- jreadey (167)
- mattjala (89)
- dependabot[bot] (32)
- bilalshaikh42 (7)
- loricooperhdf (4)
- mbruggs (3)
- adenflorian (2)
- ajelenak (2)
- rho-novatron (2)
- hyoklee (2)
- assaron (1)
- t20100 (1)
- TManhente (1)
- trmt (1)
- derobins (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 1,506 last-month
- Total dependent packages: 1
- Total dependent repositories: 1
- Total versions: 9
- Total maintainers: 1
pypi.org: hsds
Rest-based Web Service for HDF data
- Homepage: https://www.hdfgroup.org/solutions/highly-scalable-data-service-hsds/
- Documentation: https://github.com/HDFGroup/hsds/tree/master/docs
- License: Apache Software License
-
Latest release: 0.9.4
published 6 months ago
Rankings
Maintainers (1)
Dependencies
- actions/checkout v3 composite
- github/codeql-action/analyze v2 composite
- github/codeql-action/autobuild v2 composite
- github/codeql-action/init v2 composite
- actions/checkout v3 composite
- actions/dependency-review-action v1 composite
- actions/checkout v2 composite
- docker/build-push-action v2 composite
- docker/login-action v1 composite
- docker/metadata-action v3 composite
- docker/setup-buildx-action v1 composite
- docker/setup-qemu-action v1 composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- python 3.9-slim build
- hdfgroup/hsds latest
- mdsol/logspout latest
- hdfgroup/hsds latest
- hdfgroup/hsds latest
- hdfgroup/hsds latest
- hdfgroup/hsds latest
- hdfgroup/python 3.6 build
- hdfgroup/python 3.6 build
- python 3.9 build
- hdfgroup/hswritetest latest
- PyJWT *
- PyYAML *
- aiobotocore ==2.1.0
- aiofiles *
- aiohttp ==3.8.1
- aiohttp_cors *
- cryptography *
- hsds *
- numcodecs *
- numpy *
- psutil *
- pytz *
- requests-unixsocket *
- simplejson *
- aiobotocore ==2.1.0
- aiofiles ==0.8.0
- aiohttp ==3.8.1
- aiohttp-cors ==0.7.0
- aioitertools ==0.10.0
- aiosignal ==1.2.0
- async-timeout ==4.0.2
- attrs ==21.4.0
- botocore ==1.23.24
- certifi ==2022.5.18.1
- cffi ==1.15.0
- charset-normalizer ==2.0.12
- cryptography ==37.0.2
- frozenlist ==1.3.0
- hsds *
- idna ==3.3
- jmespath ==0.10.0
- multidict ==6.0.2
- numcodecs ==0.9.1
- numpy ==1.22.4
- psutil ==5.9.1
- pycparser ==2.21
- pyjwt ==2.4.0
- python-dateutil ==2.8.2
- pytz ==2022.1
- pyyaml ==6.0
- requests ==2.27.1
- requests-unixsocket ==0.3.0
- simplejson ==3.17.6
- six ==1.16.0
- urllib3 ==1.26.9
- wrapt ==1.14.1
- yarl ==1.7.2
- aiobotocore ==2.1.0
- aiofiles *
- aiohttp ==3.8.1
- aiohttp_cors *
- cryptography *
- numcodecs *
- numpy *
- psutil *
- pyjwt *
- pytz *
- pyyaml *
- requests-unixsocket *
- simplejson *
- aiobotocore == 2.5.0
- aiofiles *
- aiohttp == 3.8.5
- aiohttp_cors *
- azure-storage-blob *
- botocore *
- cryptography *
- numcodecs *
- numpy *
- psutil *
- pyjwt *
- pytz *
- pyyaml *
- requests-unixsocket *
- simplejson *