https://github.com/hosseinmoein/dataframe
C++ DataFrame for statistical, financial, and ML analysis in modern C++
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (15.6%) to scientific vocabulary
Keywords
Keywords from Contributors
Repository
C++ DataFrame for statistical, financial, and ML analysis in modern C++
Basic Info
- Host: GitHub
- Owner: hosseinmoein
- License: bsd-3-clause
- Language: C++
- Default Branch: master
- Homepage: https://hosseinmoein.github.io/DataFrame/
- Size: 47.8 MB
Statistics
- Stars: 2,793
- Watchers: 74
- Forks: 341
- Open Issues: 0
- Releases: 33
Topics
Metadata Files
README.md
DataFrame documentation with code samples
This is a C++ analytical library designed for data analysis similar to libraries in Python and R. For example, you would compare this to Pandas or R data.frame. The depth and breadth of functionalities offered by C++ DataFrame alone are greater than functionalities offered by packages such as Pandas, data.frame, and Polars combined.
You can slice the data in many different ways. You can join, merge, group-by the data. You can run various statistical, summarization, financial, and ML algorithms on the data. You can add your custom algorithms easily. You can multi-column sort, custom pick and delete the data. And more …
DataFrame also includes a large collection of analytical algorithms in form of visitors. These are from basic stats such as Mean, STDEV, Moving Averages, ... to more involved analysis such as PCA, Polynomial Fit, FFT, Eigens ... including a good collection of trading indicators. You can also easily add your own algorithms.
DataFrame also employs extensive multithreading in almost all its API’s, for large datasets. That makes DataFrame especially suitable for analyzing large datasets.
For basic operations to start you off, see Hello World and/or Cheat Sheet. For a complete list of features with code samples, see documentation.
I have followed a few principles in this library:
- Support any type either built-in or user defined without needing new code
- Never chase pointers ala linked lists, std::any, pointer to base, ...
- Have all column data in contiguous memory space
- Never use more space than you need ala unions, std::variant, ...
- Avoid copying data as much as possible
- Use multi-threading but only when it makes sense
- Do not attempt to protect the user against garbage in, garbage out
- Keep DataFrame library self-contained, meaning DataFrame must only depend on C++ language and its standard library
Performance
You have probably heard of Polars DataFrame. It is implemented in Rust and ported with zero-overhead to Python (as long as you don’t have a loop). I have been asked by many people to write a comparison for DataFrame vs. Polars. So, I finally found some time to learn a bit about Polars and write a very simple benchmark.
I wrote the following identical programs for both Polars and C++ DataFrame (and Pandas). I used Polars version: 0.19.14 (Pandas version: 1.5.3, Numpy version: 1.24.2). And I used C++23 GCC-14 compiler with -O3 option. I ran both on my, somewhat outdated, MacBook Pro (Intel chip, 96GB RAM).
In both cases, I created a dataframe with 3 random columns. The C++ DataFrame also required an additional index column of the same size. Polars doesn’t believe in index columns (that has its own pros and cons. I am not going through it here).
Each program has three identical parts. First it generates and populates 3 columns with 300m random numbers each (in case of C++ DataFrame, it must also generate a sequential index column of the same size). That is the part I am not interested in. In the second part, it calculates the mean of the first column, the variance of the second column, and the Pearson correlation of the second and third columns. In the third part, it does a select (or filter as Polars calls it) on one of the columns.
Results:
The maximum dataset I could load into Polars was 300m rows per column. Any bigger dataset blew up the memory and caused OS to kill it. I ran C++ DataFrame with 10b rows per column and I am sure it would have run with bigger datasets too. So, I was forced to run both with 300m rows to compare.
I ran each test 4 times and took the best time. Polars numbers varied a lot from one run to another, especially calculation and selection times. C++ DataFrame numbers were significantly more consistent.
| | C++ DataFrame | Polars | Pandas | | :--------------------- | ------------------------------------------------------------------------------------------------------------------: | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | Data Generation Time | 26.9459 secs | 28.4686 secs | 36.6799 secs | | Calculation Time | 1.2602 secs | 4.8766 secs | 40.3264 secs | | Selection Time | 0.4215 secs | 3.8766 secs | 8.3264 secs | | Overall Time | 28.9486 secs | 36.8763 secs | 85.8451 secs |
Owner
- Name: Hossein Moein
- Login: hosseinmoein
- Kind: user
- Location: New York
- Repositories: 19
- Profile: https://github.com/hosseinmoein
Software Engineer
GitHub Events
Total
- Create event: 11
- Release event: 2
- Issues event: 52
- Watch event: 318
- Issue comment event: 94
- Push event: 190
- Pull request review comment event: 9
- Pull request review event: 9
- Pull request event: 27
- Fork event: 30
Last Year
- Create event: 11
- Release event: 2
- Issues event: 52
- Watch event: 318
- Issue comment event: 94
- Push event: 190
- Pull request review comment event: 9
- Pull request review event: 9
- Pull request event: 27
- Fork event: 30
Committers
Last synced: over 2 years ago
Top Committers
| Name | Commits | |
|---|---|---|
| hossein.moein@kensho.com | h****n@k****m | 751 |
| Hossein Moein | 3****n | 324 |
| Hossein Moein | h****n@g****m | 181 |
| Hossein Moein | m****e@a****m | 126 |
| Justin K -Linux | j****7@g****m | 69 |
| SpaceIm | 3****m | 36 |
| bplaa-yai | j****n@3****g | 13 |
| JimmyG | j****g@g****m | 8 |
| Jernej Makovsek | j****k@g****m | 4 |
| Marcello Mansueto | m****m@m****m | 4 |
| stepan.potys | s****s@o****m | 3 |
| Julien Marrec | j****c@g****m | 2 |
| Alexandre | E****o@p****m | 1 |
| Stepan | s****s@g****m | 1 |
| Justin K -Work | J****a@i****m | 1 |
| NikBomb | n****e@g****m | 1 |
| wujinghe | w****e@q****n | 1 |
| Konstantin Sorokin | k****s@s****u | 1 |
| Moshe Schorr | m****l@g****m | 1 |
| Guillaume Jacquenot | G****t | 1 |
| Enrico | e****a@g****m | 1 |
| theirix | t****x@g****m | 1 |
| Edouard Berthe | e****e@p****m | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 146
- Total pull requests: 93
- Average time to close issues: 14 days
- Average time to close pull requests: 5 days
- Total issue authors: 91
- Total pull request authors: 14
- Average comments per issue: 4.82
- Average comments per pull request: 0.51
- Merged pull requests: 69
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 28
- Pull requests: 29
- Average time to close issues: 7 days
- Average time to close pull requests: 11 days
- Issue authors: 18
- Pull request authors: 4
- Average comments per issue: 3.43
- Average comments per pull request: 0.45
- Merged pull requests: 13
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- xkungfu (10)
- sierret (7)
- yuxiaojian01 (6)
- kburchfiel (5)
- mandeepsingh-private (3)
- yegorrr (3)
- adrian17 (3)
- shaojunjie0912 (3)
- federicomartinlara1976 (3)
- TheBlackPlague (3)
- Aratiganesh123 (3)
- YingHREN (2)
- justinjk007 (2)
- SpaceIm (2)
- thekvs (2)
Pull Request Authors
- hosseinmoein (56)
- SpaceIm (11)
- mo42 (6)
- GerHobbelt (4)
- wujinghe (3)
- andersc (3)
- OMGtechy (2)
- jchen8tw (2)
- Githubprivaxy (1)
- jmarrec (1)
- eltociear (1)
- edouardberthe (1)
- hehuaijin (1)
- Gjacquenot (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 2
- Total downloads: unknown
-
Total dependent packages: 0
(may contain duplicates) -
Total dependent repositories: 0
(may contain duplicates) - Total versions: 2
proxy.golang.org: github.com/hosseinmoein/DataFrame
- Documentation: https://pkg.go.dev/github.com/hosseinmoein/DataFrame#section-documentation
- License: bsd-3-clause
-
Latest release: v2.3.0+incompatible
published about 2 years ago
Rankings
proxy.golang.org: github.com/hosseinmoein/dataframe
- Documentation: https://pkg.go.dev/github.com/hosseinmoein/dataframe#section-documentation
- License: bsd-3-clause
-
Latest release: v2.3.0+incompatible
published about 2 years ago
Rankings
Dependencies
- actions/checkout v2 composite
- github/codeql-action/analyze v1 composite
- github/codeql-action/autobuild v1 composite
- github/codeql-action/init v1 composite