observations
Tools for loading standard data sets in machine learning
Science Score: 10.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
○codemeta.json file
-
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
2 of 5 committers (40.0%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (14.9%) to scientific vocabulary
Keywords
Repository
Tools for loading standard data sets in machine learning
Basic Info
Statistics
- Stars: 204
- Watchers: 5
- Forks: 40
- Open Issues: 22
- Releases: 0
Topics
Metadata Files
README.md
Observations
Announcement (September 16, 2018): Observations is in the process of being replaced by TensorFlow Datasets. Unlike Observations, TensorFlow Datasets is more performant, provides pipelining for >2GB data sets and all of Tensor2Tensor's, and better interfaces with
tf.data. We're working to add all features from Observations, such as its relatively simple API, supporting all of Observations' data sets, and providing a method to return NumPy arrays instead of TensorFlow Tensors.
Observations provides a one line Python API for loading standard data sets in machine learning. It automates the process from downloading, extracting, loading, and preprocessing data. Observations helps keep the workflow reproducible and follow sensible standards.
It can be used in two ways.
1. As a package
Install it.
bash
pip install observations
Import it.
```python
from observations import svhn
(xtrain, ytrain), (xtest, ytest) = svhn("~/data") ``` All functions take as input a filepath and optional preprocessing arguments. They return a tuple in the form of training data, test data, and validation data (if available). Each element in the tuple is typically a NumPy array, a tuple of NumPy arrays (e.g., features and labels), or a string (text). See the API for details.
2. As source code
Copy and paste functions inside the codebase relevant for your experiments.
```python def enwik8(path): ...
xtrain, xtest, x_valid = enwik8("~/data") ```
Each function has minimal dependencies. For example,
enwik8.py only depends on core libraries and
the external function maybe_download_and_extract in
util.py. The functions are designed to be easy to
read and hack at.
FAQ
Which approach should I take?
It depends on your use case.
- As a package, dozens of data sets are at your disposal. The package establishes sensible standards for conveniently loading in data and thus quickly experimenting with them.
- As source code, you have complete flexibility—from the initial download all the way to preprocessing the data as NumPy arrays.
How do I use minibatches of data?
The data loading functions return the full data. It's up to your needs to generate batches.
One helpful utility is
python
def generator(array, batch_size):
"""Generate batch with respect to array's first axis."""
start = 0 # pointer to where we are in iteration
while True:
stop = start + batch_size
diff = stop - array.shape[0]
if diff <= 0:
batch = array[start:stop]
start += batch_size
else:
batch = np.concatenate((array[start:], array[:diff]))
start = diff
yield batch
To use it, simply write
```python
from observations import cifar10
(xtrain, ytrain), (xtest, ytest) = cifar10("~/data")
xtraindata = generator(x_train, 256)
for batch in xtraindata: ... # operate on batch
batch = next(xtraindata) # alternatively, increment the iterator
There's also an extended version. It takes a list of arrays as input
and yields a list of batches.
python
def generator(arrays, batchsize):
"""Generate batches, one with respect to each array's first axis."""
starts = [0] * len(arrays) # pointers to where we are in iteration
while True:
batches = []
for i, array in enumerate(arrays):
start = starts[i]
stop = start + batchsize
diff = stop - array.shape[0]
if diff <= 0:
batch = array[start:stop]
starts[i] += batchsize
else:
batch = np.concatenate((array[start:], array[:diff]))
starts[i] = diff
batches.append(batch)
yield batches
To use it, simply write
python
from observations import cifar10
(xtrain, ytrain), (xtest, ytest) = cifar10("~/data")
traindata = generator([xtrain, ytrain], 256)
for xbatch, ybatch in train_data: ... # operate on batch
xbatch, ybatch = next(train_data) # alternatively, increment the iterator ```
Contributing
We'd like your help! Any pull requests which help maintain the existing functions and/or add new ones are appreciated. We follow Edward's standards for style and documentation.
Each function takes as input a filepath and optional preprocessing arguments. All necessary packages that aren't from the Python Standard Library, NumPy, or six are imported inside the function's body. The functions proceed as follows:
- Check if the extracted file(s) exist in the filepath. If it does, skip to step 4.
- Check if the compressed file(s) exist in the filepath. If it doesn't, download it.
- Extract the compressed file(s).
- Load the data into memory.
- For data sets larger than 1 GB, the function will terminate with a message advising to load the files as batches.
- Preprocess the data.
- Return a tuple in the form of training data, test data, and validation data (if available). Each element in the tuple is typically a NumPy array, a tuple of NumPy arrays (e.g., features and labels), or a string (text).
GitHub Events
Total
- Watch event: 2
Last Year
- Watch event: 2
Committers
Last synced: over 2 years ago
Top Committers
| Name | Commits | |
|---|---|---|
| Dustin Tran | d****n@g****m | 23 |
| Arvinds-ds | a****5@c****u | 5 |
| Mark van der Wilk | m****0@c****k | 1 |
| Choong Jun Jin | z****e@g****m | 1 |
| Dr. Kashif Rasul | k****l@g****m | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 11 months ago
All Time
- Total issues: 34
- Total pull requests: 20
- Average time to close issues: 8 days
- Average time to close pull requests: 2 days
- Total issue authors: 4
- Total pull request authors: 7
- Average comments per issue: 1.09
- Average comments per pull request: 2.05
- Merged pull requests: 17
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- dustinvtran (29)
- Arvinds-ds (3)
- laygr (1)
- wavelet3000 (1)
Pull Request Authors
- dustinvtran (9)
- Arvinds-ds (6)
- zepx (1)
- hajime9652 (1)
- kashif (1)
- TrellixVulnTeam (1)
- markvdw (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 557 last-month
- Total dependent packages: 1
- Total dependent repositories: 29
- Total versions: 6
- Total maintainers: 1
pypi.org: observations
Tools for loading standard data sets in machine learning
- Homepage: https://github.com/edwardlib/observations
- Documentation: https://observations.readthedocs.io/
- License: Apache License 2.0
-
Latest release: 0.1.4
published over 8 years ago
Rankings
Maintainers (1)
Dependencies
- numpy >=1.7
- six >=1.10.0