https://github.com/bytedance/mega-spark
Spark is encapsulated as a mega interface to call pandas methods
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (2.5%) to scientific vocabulary
Repository
Spark is encapsulated as a mega interface to call pandas methods
Basic Info
- Host: GitHub
- Owner: bytedance
- License: apache-2.0
- Language: Python
- Default Branch: master
- Size: 2.24 MB
Statistics
- Stars: 7
- Watchers: 3
- Forks: 5
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
Mega Spark说明
该项目旨在通过spark进行一站式数据分析与模型训练，保证最终落地的只有分析报告，可视化，以及模型训练评估结果，其次 该项目将pysaprk封装成mega对象来延续pandas的使用方法，进而实现在大数据场景下的使用pandas方法进行 数据分析和模型训练，消除相关同学在spark上投入的学习成本
目前提供以下模块：
* ml
* mega_xgboost
* sql
* megaframe
* tomega
本地安装
如果要给该项目贡献代码，在本地调试好后测试，本地安装方法
$ git clone .
$ cd megaspark
$ python install .
在线安装
建议使用官方镜像，安装最新版本。
$ pip install --index-url https://pypi.org/simple/ mega-spark
使用教程
以tomega模块为例，安装完成之后，可以使用以下方法导入使用
``` import megaspark.tomega as mg
datadf = mg.readcsv("path/to/file.csv") data_df.mega.head(5) ```
打包发布
$ cd /path/to/megaspark
$ python3 setup.py sdist bdist_wheel
$ pip3 install twine
$ python3 -m twine upload dist/*
注意事项
如果使用ml模块中的xgboost，需要在SPARK_HOME/jar中添加xgboost4j-0.72.jar以及xgboost4j-spark-0.72.jar
Owner
- Name: Bytedance Inc.
- Login: bytedance
- Kind: organization
- Location: Singapore
- Website: https://opensource.bytedance.com
- Twitter: ByteDanceOSS
- Repositories: 255
- Profile: https://github.com/bytedance
GitHub Events
Total
Last Year
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0