Science Score: 10.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
○codemeta.json file
-
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
2 of 42 committers (4.8%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.5%) to scientific vocabulary
Keywords from Contributors
alignment
flexible
closember
package-management
conda
repl
notebook
ipython
spec-0
tensor
Last synced: 11 months ago
·
JSON representation
Repository
Data Migration for the Blaze Project
Basic Info
- Host: GitHub
- Owner: blaze
- License: bsd-3-clause
- Language: Python
- Default Branch: master
- Homepage: http://odo.readthedocs.org/
- Size: 11.7 MB
Statistics
- Stars: 1,003
- Watchers: 58
- Forks: 131
- Open Issues: 218
- Releases: 0
Created over 11 years ago
· Last pushed about 4 years ago
Metadata Files
Readme
License
README.rst
Odo
===
|Build Status| |Doc Status|
.. image:: https://binstar.org/blaze/odo/badges/build.svg
:target: https://binstar.org/blaze/odo/builds
.. image:: https://binstar.org/blaze/odo/badges/version.svg
:target: https://binstar.org/blaze/odo
Data migration in Python
Documentation_
Example
-------
Odo migrates data between different containers
.. code-block:: python
>>> from odo import odo
>>> odo((1, 2, 3), list)
[1, 2, 3]
It operates on small, in-memory containers (as above) and large, out-of-core
containers (as below)
.. code-block:: python
>>> odo('myfile.hdf5::/data', 'postgresql://user:pass@host::my-table')
Table('my-table', MetaData(bind=Engine(postgresql://user:****@host)), ...)
Odo leverages the existing Python ecosystem. The example above uses
``sqlalchemy`` for SQL interation and ``h5py`` for HDF5 interaction.
Method
------
Odo migrates data using network of small data conversion functions between
type pairs. That network is below:
.. image:: https://raw.githubusercontent.com/blaze/odo/master/docs/source/images/conversions.png
:alt: odo conversions
Each node is a container type (like ``pandas.DataFrame`` or
``sqlalchemy.Table``) and each directed edge is a function that transforms or
appends one container into or onto another. We annotate these functions/edges
with relative costs.
This network approach allows ``odo`` to select the shortest path between any
two types (thank you networkx_). For performance reasons these functions often
leverage non-Pythonic systems like NumPy arrays or native ``CSV->SQL`` loading
functions. Odo is not dependent on only Python iterators.
This network approach is also robust. When libraries go missing or runtime
errors occur ``odo`` can work around these holes and find new paths.
This network approach is extensible. It is easy to write small functions and
register them to the overall graph. In the following example showing how we
convert from ``pandas.DataFrame`` to a ``numpy.ndarray``.
.. code-block:: python
from odo import convert
@convert.register(np.ndarray, pd.DataFrame, cost=1.0)
def dataframe_to_numpy(df, **kwargs):
return df.to_records(index=False)
We decorate ``convert`` functions with the target and source types as well as a
relative cost. This decoration establishes a contract that the underlying
function must fulfill, in this case with the fast ``DataFrame.to_records``
method. Similar functions exist for ``append``, to add to existing data, and
``resource`` for URI resolution.
* ``convert``: Transform dataset into new container
* ``append``: Add dataset onto existing container
* ``resource``: Given a URI find the appropriate data resource
* ``odo``: Call one of the above based on inputs.
E.g. ``odo((1, 2, 3), list) -> convert(list, (1, 2, 3))``
while ``L = []; odo((1, 2, 3), L) -> append(L, (1, 2, 3))``
Finally, ``odo`` is also aware of which containers must reside in memory and
which do not. In the graph above the *red-colored* nodes are robust to
larger-than-memory datasets. Transformations between two out-of-core datasets
operate only on the subgraph of the red nodes.
LICENSE
-------
New BSD. See `License File `__.
History
-------
Odo was factored out from the Blaze_ project.
.. _Blaze: http://blaze.pydata.org/
.. _networkx: https://networkx.github.io/
.. _Documentation: https://odo.readthedocs.org/en/latest/
.. |Build Status| image:: https://travis-ci.org/blaze/odo.png
:target: https://travis-ci.org/blaze/odo
.. |Doc Status| image:: https://readthedocs.org/projects/odo/badge/?version=latest
:target: https://readthedocs.org/projects/odo/?badge=latest
:alt: Documentation Status
Owner
- Name: Blaze
- Login: blaze
- Kind: organization
- Repositories: 15
- Profile: https://github.com/blaze
GitHub Events
Total
- Watch event: 4
Last Year
- Watch event: 4
Committers
Last synced: over 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| Phillip Cloud | c****d@g****m | 800 |
| Matthew Rocklin | m****n@g****m | 450 |
| Joe Jevnik | j****e@q****m | 82 |
| Kurt Smith | k****h@g****m | 42 |
| Blake Griffith | b****h@g****m | 28 |
| Benjamin Zaitlen | q****n@g****m | 12 |
| dhirschf | d****d@s****m | 12 |
| talumbau | t****h@c****o | 11 |
| Dan Coates | d****s@c****m | 10 |
| ywang007 | y****g@c****o | 9 |
| Jasmine Sandhu | j****u@c****o | 9 |
| Kurt Smith | k****h@c****o | 8 |
| Skipper Seabold | j****d@g****m | 6 |
| thequackdaddy | p****k@g****m | 6 |
| Eddie Hebert | e****t@q****m | 5 |
| Rami Chowdhury | r****y@c****o | 5 |
| Joe Jevnik | J****v@g****m | 4 |
| Ryan Grout | r****n@r****g | 4 |
| hussainsultan | h****z@g****m | 4 |
| root | r****t@b****) | 3 |
| Scott Sanderson | s****n@q****m | 3 |
| Gaëtan de Menten | g****n@g****m | 3 |
| Jim Crist | c****2@u****u | 3 |
| James McBride | j****r@g****m | 3 |
| Philip | b****g@g****m | 2 |
| Tom Augspurger | t****r@g****m | 2 |
| Stefan Seefeld | s****n@s****e | 2 |
| Stefan Krah | s****h@b****g | 2 |
| Christine Doig | c****t@g****m | 2 |
| timcera | t****m@c****t | 1 |
| and 12 more... | ||
Committer Domains (Top 20 + Academic)
continuum.io: 5
quantopian.com: 3
uci.edu: 1
de-valkeneer.be: 1
reback.net: 1
cerazone.net: 1
bytereef.org: 1
seefeld.name: 1
umn.edu: 1
bd37a241557b.(none): 1
ryangrout.org: 1
capitalone.com: 1
stanwell.com: 1
Issues and Pull Requests
Last synced: 11 months ago
All Time
- Total issues: 62
- Total pull requests: 38
- Average time to close issues: 7 months
- Average time to close pull requests: about 1 year
- Total issue authors: 50
- Total pull request authors: 19
- Average comments per issue: 2.85
- Average comments per pull request: 2.87
- Merged pull requests: 10
- Bot issues: 0
- Bot pull requests: 10
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- tirkarthi (3)
- ghost (3)
- rizplate (3)
- dhirschfeld (3)
- makmanalp (3)
- majidaldo (2)
- cpcloud (2)
- danielchalef (1)
- mbyim (1)
- dmkusner (1)
- blabarthe1 (1)
- rlang2017 (1)
- n0mn0m (1)
- Matttman (1)
- bwhitby (1)
Pull Request Authors
- dependabot[bot] (10)
- llllllllll (6)
- ehebert (3)
- thequackdaddy (2)
- cpcloud (2)
- kwmsmith (2)
- groutr (1)
- majidaldo (1)
- ahasha (1)
- n0mn0m (1)
- timcera (1)
- yuvallanger (1)
- michelorengo (1)
- dhirschfeld (1)
- postelrich (1)
Top Labels
Issue Labels
question (3)
user experience (3)
documentation (2)
bug (1)
windows (1)
enhancement (1)
backend (1)
feature request (1)
sql (1)
Pull Request Labels
dependencies (10)
backend (6)
enhancement (6)
testing (2)
bug (2)
easy (2)
dask (2)
sql (2)
mssql (1)
postgresql (1)
build (1)
csv (1)
pandas (1)
json (1)
Packages
- Total packages: 6
-
Total downloads:
- pypi 72,826 last-month
- Total docker downloads: 6,362
-
Total dependent packages: 7
(may contain duplicates) -
Total dependent repositories: 2,144
(may contain duplicates) - Total versions: 14
- Total maintainers: 7
pypi.org: odo
Data migration utilities
- Homepage: https://github.com/blaze/odo
- Documentation: https://odo.readthedocs.io/
- License: BSD
-
Latest release: 0.5.0
published about 10 years ago
Rankings
Dependent repos count: 0.2%
Docker downloads count: 1.1%
Stargazers count: 2.0%
Average: 2.2%
Downloads: 2.8%
Dependent packages count: 3.2%
Forks count: 4.1%
Maintainers (5)
Last synced:
11 months ago
proxy.golang.org: github.com/blaze/odo
- Documentation: https://pkg.go.dev/github.com/blaze/odo#section-documentation
- License: bsd-3-clause
-
Latest release: v0.1.0
published over 11 years ago
Rankings
Dependent packages count: 6.5%
Average: 6.7%
Dependent repos count: 6.9%
Last synced:
11 months ago
conda-forge.org: odo
- Homepage: https://github.com/blaze/odo
- License: BSD-3-Clause
-
Latest release: 0.5.1
published almost 4 years ago
Rankings
Dependent repos count: 5.1%
Dependent packages count: 10.4%
Average: 10.9%
Stargazers count: 12.8%
Forks count: 15.4%
Last synced:
11 months ago
pypi.org: odo-fixpandas20
Data migration utilities
- Homepage: https://github.com/blaze/odo
- Documentation: https://odo-fixpandas20.readthedocs.io/
- License: BSD
-
Latest release: 0.5.3
published almost 8 years ago
Rankings
Stargazers count: 2.0%
Forks count: 4.1%
Dependent packages count: 10.1%
Average: 13.2%
Dependent repos count: 21.5%
Downloads: 28.2%
Maintainers (1)
Last synced:
11 months ago
pypi.org: sangreal-odo
Data migration utilities
- Homepage: https://github.com/blaze/odo
- Documentation: https://sangreal-odo.readthedocs.io/
- License: BSD
-
Latest release: 0.0.5
published about 6 years ago
Rankings
Stargazers count: 2.0%
Forks count: 4.1%
Dependent packages count: 10.1%
Average: 16.4%
Dependent repos count: 21.5%
Downloads: 44.0%
Maintainers (1)
Last synced:
11 months ago
anaconda.org: odo
Odo is data migration for the Blaze Project. t efficiently migrates data from the source to the target through a network of conversions.
- Homepage: http://odo.pydata.org
- License: BSD-3-Clause
-
Latest release: 0.5.1
published almost 9 years ago
Rankings
Stargazers count: 24.0%
Dependent repos count: 24.6%
Forks count: 27.7%
Average: 31.9%
Dependent packages count: 51.2%
Last synced:
11 months ago
Dependencies
etc/requirements.txt
pypi
- dask >=0.11.1
- datashape >=0.5.0
- multipledispatch >=0.4.7
- networkx >=1.0
- numpy >=1.7
- pandas >=0.15.0
- toolz >=0.7.3
etc/requirements_aws.txt
pypi
- boto >=2.43.0
etc/requirements_bcolz.txt
pypi
- bcolz >=0.12.0,<1.0.0
etc/requirements_bokeh.txt
pypi
- bokeh >=0.12
etc/requirements_ci.txt
pypi
- HeapDict ==1.0.0
- Jinja2 ==2.8
- MarkupSafe ==0.23
- PyMySQL ==0.7.9
- PyYAML ==3.12
- SQLAlchemy ==1.1.3
- bcolz ==0.12.1
- bokeh ==0.12.3
- boto3 ==1.4.1
- botocore ==1.4.68
- cffi ==1.8.3
- click ==6.6
- cloudpickle ==0.2.1
- cryptography ==1.5.2
- dask ==0.11.1
- decorator ==4.0.10
- distributed ==1.13.3
- docutils ==0.12
- h5py ==2.6.0
- idna ==2.1
- jmespath ==0.9.0
- locket ==0.2.0
- mock ==2.0.0
- msgpack-python ==0.4.8
- multipledispatch ==0.4.9
- networkx ==1.11
- numexpr ==2.6.1
- numpy ==1.11.2
- pandas ==0.19.0
- paramiko ==2.0.2
- partd ==0.3.6
- psutil ==4.4.2
- psycopg2 ==2.6.2
- py ==1.4.34
- pyasn1 ==0.1.9
- pycparser ==2.17
- pymongo ==2.9.4
- pytest ==3.2.3
- python-dateutil ==2.5.3
- pytz ==2016.7
- pywebhdfs ==0.4.1
- requests ==2.11.1
- s3fs ==0.0.7
- s3transfer ==0.1.9
- sas7bdat ==2.0.7
- six ==1.10.0
- tables ==3.3.0
- tblib ==1.3.0
- toolz ==0.8.0
- tornado ==4.4.2
- zict ==0.0.3
etc/requirements_h5py.txt
pypi
- h5py >=2.6.0
etc/requirements_hdfs.txt
pypi
- pywebhdfs >=0.4.1
etc/requirements_mongo.txt
pypi
- pymongo >2.9.0,<3.0.0
etc/requirements_mysql.txt
pypi
- pymysql >=0.7
etc/requirements_postgres.txt
pypi
- psycopg2 >=2.6.0
etc/requirements_pytables.txt
pypi
- tables >=3.3.0
etc/requirements_sas.txt
pypi
- sas7bdat >=2.0.0
etc/requirements_sql.txt
pypi
- sqlalchemy >=0.8.0
etc/requirements_ssh.txt
pypi
- paramiko >=2.0.0
etc/requirements_test.txt
pypi
- pytest >=3.0.0 test
extra-requirements.txt
pypi
- bcolz *
- boto *
- h5py *
- paramiko *
- pymongo >=2.8,<3
- pywebhdfs *
- sas7bdat *
- sqlalchemy >=0.8.0
- sqlalchemy-redshift *