Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/ayaanhossain/sharedb

An on-disk pythonic embedded key-value store for compressed data storage and distributed data analysis

https://github.com/ayaanhossain/sharedb

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.7%) to scientific vocabulary

Keywords

data-analysis data-storage distributed embedded-database key-value lmb msgpack multiprocessing parallel python python-3 python-library python-script python2 python3 store
Last synced: 6 months ago · JSON representation

Repository

An on-disk pythonic embedded key-value store for compressed data storage and distributed data analysis

Basic Info
  • Host: GitHub
  • Owner: ayaanhossain
  • License: mit
  • Language: Python
  • Default Branch: master
  • Homepage:
  • Size: 1.76 MB
Statistics
  • Stars: 8
  • Watchers: 3
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Topics
data-analysis data-storage distributed embedded-database key-value lmb msgpack multiprocessing parallel python python-3 python-library python-script python2 python3 store
Created over 6 years ago · Last pushed over 1 year ago
Metadata Files
Readme License

README.md

ShareDB

CI-badge codecov-badge version-badge python-badge os-badge license-badge

ShareDB in Action • Installation • License • Contributing • Acknowledgements • API

ShareDB is a lightweight, persistent key-value store with a dictionary-like interface built on top of LMDB. It is intended to replace a python dictionary when

  1. the key-value information needs to persist locally for later reuse,
  2. the data needs to be shared across multiple processes with minimal overhead, and
  3. the keys and values can be (de)serialized via msgpack or pickle.

A ShareDB instance may be opened simultaneously in children, for reading in parallel, as long as a single process writes to the instance. Parallel writes made across processes are not safe; they are not guaranteed to be written, and may corrupt instance. ShareDB is primarily developed and tested using Linux and is compatible with both Python 2.7 and Python 3.6 and above.

ShareDB in Action

```python

from ShareDB import ShareDB # Easy import print(ShareDB.version) # Check version 1.1.4 myDB = ShareDB(path='./test.ShareDB') # Store ShareDB locally myDB['Name'] = ['Ayaan Hossain'] # Insert information myDB.get(key='Name') # Retrieve values ['Ayaan Hossain']

Accelerated batch insertion/update via a single transaction

len(myDB.multiset(kviter=zip(range(0, 10), range(10, 20))).sync()) 11 7 in myDB # Membership queries work True myDB['non-existent key'] # KeyError on invalid get as expected Traceback (most recent call last): ... KeyError: "key=non-existent key of is absent" myDB.pop(7) # Pop a key just like a dictionary 17 list(myDB.multipopitem(numitems=5)) # Or, pop as many items as you need [(0, 10), (1, 11), (2, 12), (3, 13), (4, 14)] myDB.remove(5).remove(6).length() # Chain removal of several keys 2 myDB.clear().length() # Or, clear entire ShareDB 0 myDB.drop() # Close/delete when you're done True `` ShareDBmethods either return data/result up on appropriate query, or aselfis returned to facilitate method chaining. Terminal methods.close()and.drop()` return a boolean indicating success.

Please see the /examples/ directory for full examples of ShareDB usage. Please see the API.md file for API details.

Installation

One-shot installation/upgrade of ShareDB from PyPI. bash $ pip install --upgrade ShareDB Alternatively, clone ShareDB from GitHub, bash $ git clone https://github.com/ayaanhossain/ShareDB navigate into repo, and install via pip. bash $ cd ShareDB $ pip install . You can test ShareDB with pytest inside the /tests/ directory. bash $ cd tests $ pytest Uninstallation of ShareDB is easy with pip. bash $ pip uninstall ShareDB

License

ShareDB (c) 2019-2024 Ayaan Hossain.

ShareDB is an open-source software under MIT License.

See LICENSE file for more details.

Contributing

Please discuss any issues/bugs you're facing, or any changes/features you have in mind by opening an issue, following the Contributor Covenant. See COC.md file for details. Please provide detailed information, and code snippets to facilitate debugging.

To contribute to ShareDB, please clone this repository, commit your code on a separate new branch, and submit a pull request. Please annotate and describe all new and modified code with detailed comments and new unit tests as applicable. Please ensure that modified builds pass existing unit tests before sending pull-requests. For versioning, we use SemVer.

Acknowledgements

ShareDB is maintained by:

ShareDB was originally written to meet data analysis needs in Prof. Howard Salis' Lab at Penn State University.

API

ShareDB API details can be found in the API.md file.

Owner

  • Name: Ayaan Hossain
  • Login: ayaanhossain
  • Kind: user
  • Location: Boston, MA
  • Company: Tessera Therapeutics

Computational Scientist working on Gene Editing.

GitHub Events

Total
  • Watch event: 1
  • Push event: 9
Last Year
  • Watch event: 1
  • Push event: 9

Committers

Last synced: almost 3 years ago

All Time
  • Total Commits: 195
  • Total Committers: 3
  • Avg Commits per committer: 65.0
  • Development Distribution Score (DDS): 0.179
Top Committers
Name Email Commits
Ayaan Hossain a****7@g****m 160
Ayaan Hossain a****n@u****m 33
Ayaan Hossain a****n@p****n 2
Committer Domains (Top 20 + Academic)
pop-os.localdomain: 1

Issues and Pull Requests

Last synced: 8 months ago

All Time
  • Total issues: 0
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 147 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 4
  • Total maintainers: 1
pypi.org: sharedb

An on-disk pythonic embedded key-value store for compressed data storage and distributed data analysis.

  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 147 Last month
Rankings
Dependent packages count: 10.1%
Stargazers count: 21.5%
Dependent repos count: 21.6%
Forks count: 29.8%
Average: 30.0%
Downloads: 66.9%
Maintainers (1)
ayaanhossain
Last synced: 6 months ago

Dependencies

requirements.txt pypi
  • configparser >=4.0.2
  • lmdb >=0.98
  • msgpack >=0.6.2
  • pytest-cov >=2.8.1
setup.py pypi
  • configparser >=4.0.2
  • lmdb >=0.98
  • msgpack >=0.6.2
  • pytest-cov >=2.8.1
.github/workflows/CI.yml actions
  • actions/checkout v1 composite
  • actions/setup-python v1 composite
  • codecov/codecov-action v1 composite