npy-append-array
Create Numpy .npy files by appending on the growth axis
Science Score: 44.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (13.1%) to scientific vocabulary
Repository
Create Numpy .npy files by appending on the growth axis
Basic Info
Statistics
- Stars: 64
- Watchers: 3
- Forks: 6
- Open Issues: 1
- Releases: 3
Metadata Files
README.md
NpyAppendArray
Create Numpy .npy files by appending on the growth axis (0 for C order, -1
for Fortran order). It behaves like numpy.concatenate with the difference
that the result is stored out-of-memory in a .npy file and can be reused for
further appending. After creation, the file can then be read with memory
mapping (e.g. by adding mmap_mode="r") which altogether allows to create and
read files (optionally) larger than the machine's main memory.
Some possible applications:
1. efficiently create large .npy (optionally database-like) files
* Handling of offsets not included, can be done in an extra array
* Large legacy files can be made appendable by calling ensure_appendable
* can (optionally) be performed in-place to minimize disk space usage
2. create binary log files (optionally on low-memory embedded devices)
* Check the option rewrite_header_on_append=False for extra efficiency
* Binary log files can be accessed very efficiently without parsing
* Incomplete files can be recovered efficiently by calling recover
Another feature of this library is the (above mentioned) recover function,
which makes incomplete .npy files readable by numpy.load again, no matter
whether they should be appended to or not.
Incomplete files can be the result of broken downloads or unfinished writes.
Recovery works by rewriting the header and inferring the growth axis (see
above) by the file size. As the data length may not be evenly divisible by the
non-append-axis shape, incomplete entries can either be ignored
(zerofill_incomplete=False), which probably makes sense in most scenarios.
Alternatively, to squeeze out the as much information from the file as
possible, zerofill_incomplete=True can be used, which fills the incomplete
last append axis item with zeros.
Raises ValueError instead of TypeError since version 0.9.14 to be more
consistent with Numpy.
NpyAppendArray can be used in multithreaded environments.
Installation
bash
conda install -c conda-forge npy-append-array
or
bash
pip install npy-append-array
Usage
```python from npyappendarray import NpyAppendArray import numpy as np
arr1 = np.array([[1,2],[3,4]]) arr2 = np.array([[1,2],[3,4],[5,6]])
filename = 'out.npy'
with NpyAppendArray(filename, deleteifexists=True) as npaa: npaa.append(arr1) npaa.append(arr2) npaa.append(arr2)
data = np.load(filename, mmap_mode="r")
print(data) ```
Concurrency
Concurrency can be achieved by multithreading: A single NpyAppendArray
object (per file) needs to be created. Then, append can be called from
multiple threads and locks will ensure that file writes do not happen in
parallel. When using with a with statement, make sure the join happens
within it, compare test.py.
Multithreaded writes are not the pinnacle of what is technically possible with
modern operating systems. It would be highly desirable to use async file
writes. However, although modules like aiofile exist, this is currently not
supported natively by Python or Numpy, compare
https://github.com/python/cpython/issues/76742
Implementation Details
NpyAppendArray contains a modified, partial version of format.py from the
Numpy package. It ensures that array headers are created with 21
(=len(str(8*2**64-1))) bytes of spare space. This allows to fit an array of
maxed out dimensions (for a 64 bit machine) without increasing the array
header size. This allows to simply rewrite the header as we append data to the
end of the .npy file.
Supported Systems
Tested with Ubuntu Linux, macOS and Windows.
Owner
- Name: Michael
- Login: xor2k
- Kind: user
- Repositories: 3
- Profile: https://github.com/xor2k
Citation (citation.cff)
abstract: Create Numpy .npy files by appending on the growth axis authors: - family-names: Siebert given-names: Michael orcid: 0000-0002-1369-6321 - family-names: Adelman given-names: Joshua - family-names: Git given-names: Yoav cff-version: 1.2.0 date-released: '2023-02-24' doi: 10.5281/zenodo.13820984 license: - mit repository-code: https://github.com/xor2k/npy-append-array/tree/0.9.16 title: 'xor2k/npy-append-array: 0.9.16' type: software version: 0.9.16
GitHub Events
Total
- Create event: 1
- Release event: 3
- Issues event: 4
- Watch event: 3
- Issue comment event: 2
- Push event: 3
Last Year
- Create event: 1
- Release event: 3
- Issues event: 4
- Watch event: 3
- Issue comment event: 2
- Push event: 3
Committers
Last synced: almost 3 years ago
Top Committers
| Name | Commits | |
|---|---|---|
| Michael Siebert | m****k@g****m | 25 |
| Joshua Adelman | s****s@u****m | 1 |
| gityoav | 3****v@u****m | 1 |
Issues and Pull Requests
Last synced: 8 months ago
All Time
- Total issues: 10
- Total pull requests: 3
- Average time to close issues: 3 months
- Average time to close pull requests: 8 months
- Total issue authors: 10
- Total pull request authors: 2
- Average comments per issue: 2.3
- Average comments per pull request: 3.67
- Merged pull requests: 2
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 5
- Pull requests: 0
- Average time to close issues: about 2 months
- Average time to close pull requests: N/A
- Issue authors: 5
- Pull request authors: 0
- Average comments per issue: 1.2
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- Matmad-B (1)
- scsanty (1)
- Ahaeflig (1)
- relativityspace-jsmith (1)
- tovrstra (1)
- oneil571 (1)
- sytest7 (1)
- zhuwenxing (1)
- pdmourao (1)
Pull Request Authors
- gityoav (2)
- synapticarbors (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 2
-
Total downloads:
- pypi 40,524 last-month
- Total docker downloads: 26,943
-
Total dependent packages: 2
(may contain duplicates) -
Total dependent repositories: 24
(may contain duplicates) - Total versions: 26
- Total maintainers: 1
pypi.org: npy-append-array
Create Numpy .npy files by appending on the growth axis
- Homepage: https://github.com/xor2k/npy-append-array
- Documentation: https://npy-append-array.readthedocs.io/
- License: mit
-
Latest release: 0.9.19
published 9 months ago
Rankings
Maintainers (1)
conda-forge.org: npy-append-array
Create Numpy `.npy` files that are larger than the main memory by appending on the zero axis.
- Homepage: https://github.com/xor2k/npy-append-array
- License: MIT
-
Latest release: 0.9.13
published over 4 years ago