https://github.com/vaexio/vaex
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
Science Score: 36.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
6 of 74 committers (8.1%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.6%) to scientific vocabulary
Keywords
Keywords from Contributors
Repository
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
Basic Info
- Host: GitHub
- Owner: vaexio
- License: mit
- Language: Python
- Default Branch: master
- Homepage: https://vaex.io
- Size: 133 MB
Statistics
- Stars: 8,422
- Watchers: 139
- Forks: 600
- Open Issues: 550
- Releases: 4
Topics
Metadata Files
README.md
What is Vaex?
Vaex is a high performance Python library for lazy Out-of-Core DataFrames
(similar to Pandas), to visualize and explore big tabular datasets. It
calculates statistics such as mean, sum, count, standard deviation etc, on an
N-dimensional grid for more than a billion (10^9) samples/rows per
second. Visualization is done using histograms, density plots and 3d
volume rendering, allowing interactive exploration of big data. Vaex uses
memory mapping, zero memory copy policy and lazy computations for best
performance (no memory wasted).
Installing
With pip:
$ pip install vaex
Or conda:
$ conda install -c conda-forge vaex
For more details, see the documentation
Key features
Instant opening of Huge data files (memory mapping)
HDF5 and Apache Arrow supported.
Read the documentation on how to efficiently convert your data from CSV files, Pandas DataFrames, or other sources.
Lazy streaming from S3 supported in combination with memory mapping.
Expression system
Don't waste memory or time with feature engineering, we (lazily) transform your data when needed.
Out-of-core DataFrame
Filtering and evaluating expressions will not waste memory by making copies; the data is kept untouched on disk, and will be streamed only when needed. Delay the time before you need a cluster.
Fast groupby / aggregations
Vaex implements parallelized, highly performant groupby operations, especially when using categories (>1 billion/second).
Fast and efficient join
Vaex doesn't copy/materialize the 'right' table when joining, saving gigabytes of memory. With subsecond joining on a billion rows, it's pretty fast!
More features
- Remote DataFrames (documentation coming soon)
- Integration into Jupyter and Voila for interactive notebooks and dashboards
- Machine Learning without (explicit) pipelines
Contributing
See contributing page.
Slack
Join the discussion in our Slack channel!
Learn more about Vaex
Articles
- Beyond Pandas: Spark, Dask, Vaex and other big data technologies battling head to head (includes benchmarks)
- 7 reasons why I love Vaex for data science (tips and trics)
- ML impossible: Train 1 billion samples in 5 minutes on your laptop using Vaex and Scikit-Learn
- How to analyse 100 GB of data on your laptop with Python
- Flying high with Vaex: analysis of over 30 years of flight data in Python
- Vaex: A DataFrame with super strings - Speed up your text processing up to a 1000x
- Vaex: Out of Core Dataframes for Python and Fast Visualization - 1 billion row datasets on your laptop
Watch our more recent talks:
Contact us for data science solutions, training, or enterprise support at https://vaex.io/
Owner
- Name: vaex io
- Login: vaexio
- Kind: organization
- Email: contact@vaex.io
- Location: the Netherlands
- Website: https://vaex.io
- Repositories: 13
- Profile: https://github.com/vaexio
Big data made simple. Visualization and exploration. Machine learning and deployment.
GitHub Events
Total
- Issues event: 12
- Watch event: 198
- Issue comment event: 60
- Push event: 2
- Pull request review comment event: 7
- Pull request review event: 10
- Pull request event: 11
- Fork event: 14
- Create event: 1
Last Year
- Issues event: 12
- Watch event: 198
- Issue comment event: 60
- Push event: 2
- Pull request review comment event: 7
- Pull request review event: 10
- Pull request event: 11
- Fork event: 14
- Create event: 1
Committers
Last synced: 9 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Maarten A. Breddels | m****s@g****m | 2,889 |
| Jovan Veljanoski | j****i@g****m | 299 |
| Jovan | jv@r****k | 31 |
| Bulat Yaminov | b****v@g****m | 10 |
| Ben Epstein | b****n@w****u | 8 |
| xdssio | 3****o | 7 |
| Kyle McEntush | s****s@g****m | 6 |
| shareactor | y****n@s****o | 5 |
| ddelange | 1****e | 5 |
| Steven Rieder | s****n@r****l | 5 |
| Matthew Barber | q****t@g****m | 4 |
| Kyle McEntush | k****h@i****m | 4 |
| Nick Crews | n****s@g****m | 3 |
| Sai Kiran | n****3@g****m | 3 |
| Thomas Delteil | t****i@m****m | 3 |
| marload | r****8@g****m | 3 |
| Naohiro Heya | d****b@g****m | 3 |
| franz.media | f****r@g****m | 3 |
| yohplala | y****a | 2 |
| Christian Laforte | c****e@a****m | 2 |
| Chiao | c****n@g****m | 2 |
| Dougal J. Sutherland | d****l@g****m | 2 |
| Eduardo Balbinot | e****t@g****m | 2 |
| Franz Wöllert | f****t@g****m | 2 |
| Marco Paolini | m****i@g****m | 2 |
| Meredith Durbin | m****n@g****m | 2 |
| Ralf Gommers | r****s@g****m | 2 |
| fsiola | f****a@g****m | 2 |
| Alex V. Kotlar | a****r@b****u | 1 |
| Alenka Frim | A****F | 1 |
| and 44 more... | ||
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 5 months ago
All Time
- Total issues: 154
- Total pull requests: 137
- Average time to close issues: 5 months
- Average time to close pull requests: 3 months
- Total issue authors: 115
- Total pull request authors: 26
- Average comments per issue: 2.38
- Average comments per pull request: 3.12
- Merged pull requests: 17
- Bot issues: 0
- Bot pull requests: 1
Past Year
- Issues: 19
- Pull requests: 23
- Average time to close issues: 2 days
- Average time to close pull requests: 2 days
- Issue authors: 16
- Pull request authors: 5
- Average comments per issue: 0.26
- Average comments per pull request: 3.52
- Merged pull requests: 10
- Bot issues: 0
- Bot pull requests: 1
Top Authors
Issue Authors
- Ben-Epstein (8)
- NickCrews (8)
- ashsharma96 (6)
- mfouesneau (4)
- schwingkopf (3)
- honno (3)
- myloe00 (2)
- DougRzz (2)
- grafail (2)
- ddelange (2)
- iisakkirotko (2)
- khanfarhan10 (2)
- meta-ks (2)
- Piyush23Rai (2)
- vignesh-bungee (2)
Pull Request Authors
- maartenbreddels (40)
- JovanVeljanoski (34)
- ddelange (15)
- xdssio (11)
- Ben-Epstein (7)
- EwoutH (4)
- 2maz (3)
- NickCrews (3)
- Shashank1202 (2)
- mgorny (2)
- ghost (1)
- And0k (1)
- AlenkaF (1)
- detayotella (1)
- jaegglic (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 8
-
Total downloads:
- pypi 43,616 last-month
- Total docker downloads: 13,450
-
Total dependent packages: 40
(may contain duplicates) -
Total dependent repositories: 150
(may contain duplicates) - Total versions: 187
- Total maintainers: 1
pypi.org: vaex
Out-of-Core DataFrames to visualize and explore big tabular datasets
- Homepage: https://www.github.com/vaexio/vaex
- Documentation: https://vaex.readthedocs.io/
- License: MIT
-
Latest release: 4.17.0
published over 2 years ago
Rankings
Maintainers (1)
pypi.org: vaex-ml
Machine learning support for vaex
- Homepage: https://www.github.com/vaexio/vaex
- Documentation: https://vaex-ml.readthedocs.io/
- License: MIT
-
Latest release: 0.18.3
published over 2 years ago
Rankings
Maintainers (1)
pypi.org: vaex-graphql
GraphQL support for accessing vaex DataFrame
- Homepage: https://www.github.com/vaexio/vaex
- Documentation: https://vaex-graphql.readthedocs.io/
- License: MIT
-
Latest release: 0.2.0
published almost 5 years ago
Rankings
Maintainers (1)
proxy.golang.org: github.com/vaexio/vaex
- Documentation: https://pkg.go.dev/github.com/vaexio/vaex#section-documentation
- License: mit
-
Latest release: v0.1.8
published over 11 years ago
Rankings
conda-forge.org: vaex-core
- Homepage: https://www.github.com/vaexio/vaex
- License: MIT
-
Latest release: 4.14.0
published over 3 years ago
Rankings
conda-forge.org: vaex-viz
- Homepage: https://www.github.com/vaexio/vaex
- License: MIT
-
Latest release: 0.5.4
published over 3 years ago
Rankings
pypi.org: vaex-contrib
Community contributed modules to vaex
- Homepage: https://www.github.com/vaexio/vaex
- Documentation: https://vaex-contrib.readthedocs.io/
- License: MIT
-
Latest release: 0.1.3
published about 3 years ago
Rankings
Maintainers (1)
conda-forge.org: vaex-ml
Wrappers for various machine learning libraries to make them integrate into vaex.
- Homepage: https://www.github.com/vaexio/vaex
- License: MIT
-
Latest release: 0.18.0
published over 3 years ago
Rankings
Dependencies
- ./ci/actions/windll * composite
- actions/cache v2 composite
- actions/checkout v2 composite
- actions/download-artifact v2 composite
- actions/upload-artifact v2 composite
- google-github-actions/setup-gcloud v0 composite
- ifaxity/wait-on-action v1 composite
- mamba-org/provision-with-micromamba main composite
- maxim-lobanov/setup-xcode v1 composite
- actions/checkout v1 composite
- actions/setup-python v2 composite
- actions/upload-artifact v1 composite
- ./ci/actions/windll * composite
- actions/checkout v1 composite
- actions/setup-python v2 composite
- actions/upload-artifact v1 composite
- healpy *
- scipy *
- vaex ==3.0.0