Data

Tools

Indexes

Applications

Experiments

Open Source Science

persia

High performance distributed framework for training deep learning recommendation models based on PyTorch.

https://github.com/persiaml/persia

Science Score: 54.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.4%) to scientific vocabulary

Keywords

bagua deep-learning distributed-computing machine-learning persia pytorch recommender-system rust-lang

Keywords from Contributors

meshing standardization pipeline-testing datacleaner data-profilers pde pinn interpretability interactive differentiation
Last synced: 4 months ago · JSON representation ·

Repository

High performance distributed framework for training deep learning recommendation models based on PyTorch.

Basic Info
  • Host: GitHub
  • Owner: PersiaML
  • License: mit
  • Language: Rust
  • Default Branch: main
  • Homepage:
  • Size: 1.06 MB
Statistics
  • Stars: 408
  • Watchers: 8
  • Forks: 55
  • Open Issues: 0
  • Releases: 0
Topics
bagua deep-learning distributed-computing machine-learning persia pytorch recommender-system rust-lang
Created over 4 years ago · Last pushed 7 months ago
Metadata Files
Readme License Citation

README.md

tutorials Documentation Status PyPI version PyPI downloads Docker Pulls license

WARNING: THIS PROJECT IS CURRENTLY NOT MAINTAINED, DUE TO COMPANY REORGANIZATION.

PERSIA (Parallel rEcommendation tRaining System with hybrId Acceleration) is developed by AI platform@Kuaishou Technology, collaborating with ETH. It is a PyTorch-based (the first public one to our best knowledge) system for training large scale deep learning recommendation models on commodity hardwares. It is capable of training recommendation models with up to 100 trillion parameters. To the best of our knowledge, this is the largest model size in recommendation systems so far. Empirical study on public datasets indicate PERSIA's significant advantage over several other existing training systems in recommendation [1]. Its efficiency and robustness have also been validated by multiple applications with 100 million level DAU at Kuaishou.

Disclaimer: The program is usable and has served several important businesses. However, the official English documentation and tutorials are still under heavy construction and they are a bit raw now. We encourage adventurers to try out PERSIA and contribute!

News

Links

References

  1. Xiangru Lian, Binhang Yuan, Xuefeng Zhu, Yulong Wang, Yongjun He, Honghuan Wu, Lei Sun, Haodong Lyu, Chengjun Liu, Xing Dong, Yiqiao Liao, Mingnan Luo, Congfei Zhang, Jingru Xie, Haonan Li, Lei Chen, Renjie Huang, Jianying Lin, Chengchun Shu, Xuezhong Qiu, Zhishan Liu, Dongying Kong, Lei Yuan, Hai Yu, Sen Yang, Ce Zhang, & Ji Liu. (2021). Persia: A Hybrid System Scaling Deep Learning Based Recommenders up to 100 Trillion Parameters.

  2. Ji Liu & Ce Zhang. (2021). Distributed Learning Systems with First-order Methods.

License

This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.

Owner

  • Name: PersiaML
  • Login: PersiaML
  • Kind: organization

Citation (CITATION.cff) 

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "Lian"
  given-names: "Xiangru"
  orcid: "https://orcid.org/0000-0003-4456-8127"
- family-names: "Yuan"
  given-names: "Binhang"
- family-names: "Zhu"
  given-names: "Xuefeng"
- family-names: "Wang"
  given-names: "Yulong"
- family-names: "He"
  given-names: "Yongjun"
- family-names: "Wu"
  given-names: "Honghuan"
- family-names: "Sun"
  given-names: "Lei"
- family-names: "Lyu"
  given-names: "Haodong"
- family-names: "Liu"
  given-names: "Chengjun"
- family-names: "Dong"
  given-names: "Xing"
- family-names: "Liao"
  given-names: "Yiqiao"
- family-names: "Luo"
  given-names: "Mingnan"
- family-names: "Zhang"
  given-names: "Congfei"
- family-names: "Xie"
  given-names: "Jingru"
- family-names: "Li"
  given-names: "Haonan"
- family-names: "Chen"
  given-names: "Lei"
- family-names: "Huang"
  given-names: "Renjie"
- family-names: "Lin"
  given-names: "Jianying"
- family-names: "Shu"
  given-names: "Chengchun"
- family-names: "Qiu"
  given-names: "Xuezhong"
- family-names: "Liu"
  given-names: "Zhishan"
- family-names: "Kong"
  given-names: "Dongying"
- family-names: "Yuan"
  given-names: "Lei"
- family-names: "Yu"
  given-names: "Hai"
- family-names: "Yang"
  given-names: "Sen"
- family-names: "Zhang"
  given-names: "Ce"
- family-names: "Liu"
  given-names: "Ji"
title: "Persia: A Hybrid System Scaling Deep Learning Based Recommenders up to 100 Trillion Parameters"
date-released: 2021-11-12
url: "https://github.com/PersiaML/Persia"

GitHub Events

Total
  • Issues event: 2
  • Watch event: 16
  • Delete event: 44
  • Issue comment event: 122
  • Pull request event: 86
  • Fork event: 3
  • Create event: 42
Last Year
  • Issues event: 2
  • Watch event: 16
  • Delete event: 44
  • Issue comment event: 122
  • Pull request event: 86
  • Fork event: 3
  • Create event: 42

Committers

Last synced: 7 months ago

All Time
  • Total Commits: 247
  • Total Committers: 7
  • Avg Commits per committer: 35.286
  • Development Distribution Score (DDS): 0.648
Past Year
  • Commits: 2
  • Committers: 1
  • Avg Commits per committer: 2.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Xiangru Lian a****n@m****m 87
Yulong Wang w****r 62
dependabot[bot] 4****] 49
Xuefeng Zhu 3****z 42
Ji Liu j****c@g****m 4
Jingru Xie x****o@g****m 2
github-actions[bot] 4****] 1
Committer Domains (Top 20 + Academic)
mail.xrlian.com: 1

Issues and Pull Requests

Last synced: 4 months ago

All Time
  • Total issues: 1
  • Total pull requests: 288
  • Average time to close issues: 18 days
  • Average time to close pull requests: 8 days
  • Total issue authors: 1
  • Total pull request authors: 1
  • Average comments per issue: 3.0
  • Average comments per pull request: 2.69
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 288
Past Year
  • Issues: 1
  • Pull requests: 87
  • Average time to close issues: 18 days
  • Average time to close pull requests: 9 days
  • Issue authors: 1
  • Pull request authors: 1
  • Average comments per issue: 3.0
  • Average comments per pull request: 2.38
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 87
View more stats
Top Authors
Issue Authors
  • lwz23 (1)
Pull Request Authors
  • dependabot[bot] (341)
Top Labels
Issue Labels
wontfix (1)
Pull Request Labels
dependencies (341) wontfix (264) rust (35)