Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/dineshpinto/data-aggregator

A high performance solution to analysing millions of incoming data points using JIT, vectorisation and C arrays.

https://github.com/dineshpinto/data-aggregator

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (8.1%) to scientific vocabulary

Keywords

data-science high-performance numpy python
Last synced: 4 months ago · JSON representation

Repository

A high performance solution to analysing millions of incoming data points using JIT, vectorisation and C arrays.

Basic Info
  • Host: GitHub
  • Owner: dineshpinto
  • License: mit
  • Language: Jupyter Notebook
  • Default Branch: main
  • Homepage:
  • Size: 1.29 MB
Statistics
  • Stars: 2
  • Watchers: 2
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Topics
data-science high-performance numpy python
Created almost 4 years ago · Last pushed almost 4 years ago
Metadata Files
Readme License

README.md

data_aggregator

A Python solution to the Messari challenge using a combination of array vectorization, JIT compilation and C-arrays to improve performance.

Files

Detailed notes are given in the individual files.

File 1: dataaggregatorlogic_tests.ipynb

Here we test out the logic on simulated data points, and benchmark the performance of the algorithm.

Link to file 1

File 2: dataaggregatormain.ipynb

Here the logic is used by executing the binary file and analyzing its input in real-time.

Link to file 2

Further improvements

  • Compile Numba against the Intel SVML libraries for further performance enhancement (currently not possible on Mac M1 arch)
  • Added parallel processing in Numba for processing distinct arrays concurrently (should be relatively trivial, but may require dropping down to ctypes arrays to ensure processes can write concurrently)

Notes

The conda environment contains a lot of additional libraries for visualization, notebooks etc. The logic itself only requires a minimal subset of those libraries (namely numpy and numba).

Owner

  • Name: Dinesh Pinto
  • Login: dineshpinto
  • Kind: user
  • Location: Switzerland/Germany

quantum info PhD student @ EPFL, pythonista & rustacean

GitHub Events

Total
  • Watch event: 1
Last Year
  • Watch event: 1

Issues and Pull Requests

Last synced: 11 months ago

All Time
  • Total issues: 0
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels