https://github.com/lebedov/msgpack-numpy
Serialize numpy arrays using msgpack
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
1 of 10 committers (10.0%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (14.1%) to scientific vocabulary
Keywords
Repository
Serialize numpy arrays using msgpack
Basic Info
Statistics
- Stars: 205
- Watchers: 9
- Forks: 31
- Open Issues: 4
- Releases: 0
Topics
Metadata Files
README.md
Numpy Data Type Serialization Using Msgpack
Package Description
This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types provided by numpy using the highly efficient msgpack format. Serialization of Python's native complex data types is also supported.
Installation
msgpack-numpy requires msgpack-python and numpy. If you have pip installed on your system, run
pip install msgpack-numpy
to install the package and all dependencies. You can also download the source tarball, unpack it, and run
python setup.py install
from within the source directory.
Usage
The easiest way to use msgpack-numpy is to call its monkey patching function after importing the Python msgpack package:
import msgpack
import msgpack_numpy as m
m.patch()
This will automatically force all msgpack serialization and deserialization routines (and other packages that use them) to become numpy-aware. Of course, one can also manually pass the encoder and decoder provided by msgpack-numpy to the msgpack routines:
import msgpack
import msgpack_numpy as m
import numpy as np
x = np.random.rand(5)
x_enc = msgpack.packb(x, default=m.encode)
x_rec = msgpack.unpackb(x_enc, object_hook=m.decode)
msgpack-numpy will try to use the binary (fast) extension in msgpack by default.
If msgpack was not compiled with Cython (or if the MSGPACK_PUREPYTHON
variable is set), it will fall back to using the slower pure Python msgpack
implementation.
Notes
The primary design goal of msgpack-numpy is ensuring preservation of numerical data types during msgpack serialization and deserialization. Inclusion of type information in the serialized data necessarily incurs some storage overhead; if preservation of type information is not needed, one may be able to avoid some of this overhead by writing a custom encoder/decoder pair that produces more efficient serializations for those specific use cases.
Numpy arrays with a dtype of 'O' are serialized/deserialized using pickle as a fallback solution to enable msgpack-numpy to handle such arrays. As the additional overhead of pickle serialization negates one of the reasons to use msgpack, it may be advisable to either write a custom encoder/decoder to handle the specific use case efficiently or else not bother using msgpack-numpy.
Note that numpy arrays deserialized by msgpack-numpy are read-only and must be copied if they are to be modified.
Development
The latest source code can be obtained from GitHub.
msgpack-numpy maintains compatibility with python versions 2.7 and 3.5+.
Install tox to support testing
across multiple python versions in your development environment. If you
use conda to install python use
tox-conda to automatically manage
testing across all supported python versions.
# Using a system python
pip install tox
# Additionally, using a conda-provided python
pip install tox tox-conda
Execute tests across supported python versions:
tox
Authors
See the included AUTHORS.md file for more information.
License
This software is licensed under the BSD License. See the included LICENSE.md file for more information.
See Also
msgpack-numpy-js by Egil Möller is a JavaScript library that can serialize and deserialize JS typed arrays with msgpack.
msgpack-numpy-rs by Terrence Liu is a Rust crate (mostly) compatible with msgpack-numpy that can serialize and deserialize numpy arrays with msgpack.
Owner
- Name: Lev E. Givon
- Login: lebedov
- Kind: user
- Location: Greater Pittsburgh Area
- Company: Janssen R&D
- Website: https://lebedov.github.io
- Repositories: 33
- Profile: https://github.com/lebedov
Senior Data Scientist / Machine Learning Researcher
GitHub Events
Total
- Watch event: 10
- Issue comment event: 2
- Fork event: 1
Last Year
- Watch event: 10
- Issue comment event: 2
- Fork event: 1
Committers
Last synced: 9 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Lev E. Givon | l****v@c****u | 102 |
| Alex Ford | a****d@a****m | 4 |
| John Tyree | j****e@g****m | 2 |
| Your Name | c****s@g****m | 1 |
| P.E. Viau | v****e@g****m | 1 |
| Kolten Pearson | k****n@g****m | 1 |
| Jungkook Park | p****a@g****m | 1 |
| Jose Tiago Macara Coutinho | c****o@g****m | 1 |
| Colin Jermain | c****n@g****m | 1 |
| Lev E. Givon | l****n@d****m | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 45
- Total pull requests: 11
- Average time to close issues: about 2 months
- Average time to close pull requests: about 2 months
- Total issue authors: 39
- Total pull request authors: 10
- Average comments per issue: 2.22
- Average comments per pull request: 1.55
- Merged pull requests: 9
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 3
- Pull requests: 0
- Average time to close issues: 3 days
- Average time to close pull requests: N/A
- Issue authors: 3
- Pull request authors: 0
- Average comments per issue: 1.67
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- hmeine (2)
- markusr (2)
- dineshbvadhia (2)
- KennyChenBasis (2)
- ppwwyyxx (2)
- e0en (2)
- thatGreekGuy96 (1)
- nikhil8786 (1)
- goodboy (1)
- liran-funaro (1)
- tvkpz (1)
- subiol (1)
- DomHudson (1)
- valmar (1)
- pklapperich (1)
Pull Request Authors
- asford (2)
- pjknkda (1)
- crispamares (1)
- johntyree (1)
- cjermain (1)
- KennyChenBasis (1)
- blen2r (1)
- tiagocoutinho (1)
- koltenpearson (1)
- econtal (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 4
-
Total downloads:
- pypi 956,216 last-month
- Total docker downloads: 24,580,383
-
Total dependent packages: 73
(may contain duplicates) -
Total dependent repositories: 1,803
(may contain duplicates) - Total versions: 51
- Total maintainers: 2
pypi.org: msgpack-numpy
Numpy data serialization using msgpack
- Homepage: https://github.com/lebedov/msgpack-numpy
- Documentation: https://msgpack-numpy.readthedocs.io/
- License: BSD
-
Latest release: 0.4.8
published over 3 years ago
Rankings
Maintainers (1)
spack.io: py-msgpack-numpy
This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types provided by numpy using the highly efficient msgpack format. Serialization of Python's native complex data types is also supported.
- Homepage: https://github.com/lebedov/msgpack-numpy
- License: []
-
Latest release: 0.4.7
published almost 4 years ago
Rankings
Maintainers (1)
conda-forge.org: msgpack-numpy
This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types provided by numpy using the highly efficient msgpack format. Serialization of Python's native complex data types is also supported.
- Homepage: https://github.com/lebedov/msgpack-numpy
- License: BSD-3-Clause
-
Latest release: 0.4.8
published over 3 years ago
Rankings
anaconda.org: msgpack-numpy
This package provides encoding and decoding routines that enable the serialization and deserialization of numerical and array data types provided by numpy using the highly efficient msgpack format. Serialization of Python's native complex data types is also supported.
- Homepage: https://github.com/lebedov/msgpack-numpy
- License: BSD-3-Clause
-
Latest release: 0.4.8
published 7 months ago
Rankings
Dependencies
- msgpack >=0.5.2
- numpy >=1.9.0
- msgpack >=0.5.2
- numpy >=1.9.0