https://github.com/bsc-wdc/dislib

The Distributed Computing library for python implemented using PyCOMPSs programming model for HPC.

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
○
Academic publication links
○
Committers with academic emails
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (15.9%) to scientific vocabulary

Keywords

big-data distributed-computing hpc machine-learning python

Keywords from Contributors

pipeline-framework singularity slurm workflow-management-system workflows

Last synced: 5 months ago · JSON representation

Repository

The Distributed Computing library for python implemented using PyCOMPSs programming model for HPC.

Basic Info

Host: GitHub
Owner: bsc-wdc
License: apache-2.0
Language: Python
Default Branch: master
Homepage:
Size: 5.91 MB

Statistics

Stars: 50
Watchers: 6
Forks: 25
Open Issues: 47
Releases: 17

Topics

big-data distributed-computing hpc machine-learning python

Created over 7 years ago · Last pushed 9 months ago

Metadata Files

Readme Changelog Contributing License

Distributed computing library implemented over PyCOMPSs programming model for HPC.

Website • Documentation • Releases • Slack

Introduction

The Distributed Computing Library (dislib) provides distributed algorithms ready to use as a library. So far, dislib is highly focused on machine learning algorithms, and it is greatly inspired by scikit-learn. However, other types of numerical algorithms might be added in the future. The library has been implemented on top of PyCOMPSs programming model, and it is being developed by the Workflows and Distributed Computing group of the Barcelona Supercomputing Center. dislib allows easy local development through docker. Once the code is finished, it can be run directly on any distributed platform without any further changes. This includes clusters, supercomputers, clouds, and containerized platforms.

Introduction
Contents
Quickstart
Availability
Contributing
Citing dislib
- Bibtex:
Acknowledgements
License

Quickstart

Get started with dislib following our quickstart guide.

Availability

Currently, the following supercomputers have already PyCOMPSs installed and ready to use. If you need help configuring your own cluster or supercomputer, drop us an email and we will be pleased to help.

Marenostrum 4 - Barcelona Supercomputing Center (BSC)
Minotauro - Barcelona Supercomputing Center (BSC)
Nord 3 - Barcelona Supercomputing Center (BSC)
Cobi - Barcelona Supercomputing Center (BSC)
Juron - Jülich Supercomputing Centre (JSC)
Jureca - Jülich Supercomputing Centre (JSC)
Ultraviolet - The Genome Analysis Center (TGAC)
Archer - University of Edinburgh’s Advanced Computing Facility (ACF)
Axiom - University of Novi Sad, Faculty of Sciences (UNSPMF)

Supported architectures: - Intel SSF architectures - IBM's Power 9

Contributing

Contributions are welcome and very much appreciated. We are also open to starting research collaborations or mentoring if you are interested in or need assistance implementing new algorithms. Please refer to our Contribution Guide for more details.

Citing dislib

If you use dislib in a scientific publication, we would appreciate you citing the following paper:

J. Álvarez Cid-Fuentes, S. Solà, P. Álvarez, A. Castro-Ginard, and R. M. Badia, "dislib: Large Scale High Performance Machine Learning in Python," in Proceedings of the 15th International Conference on eScience, 2019, pp. 96-105

Bibtex:

latex @inproceedings{dislib, title = {{dislib: Large Scale High Performance Machine Learning in Python}}, author = {Javier Álvarez Cid-Fuentes and Salvi Solà and Pol Álvarez and Alfred Castro-Ginard and Rosa M. Badia}, booktitle = {Proceedings of the 15th International Conference on eScience}, pages = {96-105}, year = {2019}, }

Acknowledgements

This work has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement H2020-MSCA-COFUND-2016-754433.

This work has also received funding from the collaboration project between the Barcelona Supercomputing Center (BSC) and Fujitsu Ltd.

In addition, the development of this software has been also supported by the following institutions:

Spanish Government under contracts SEV2015-0493, TIN2015-65316 and PID2019-107255G.
Generalitat de Catalunya under contract 2017-SGR-01414 and the CECH project, co-funded with 50% by the European Regional Development Fund under the framework of the ERFD Operative Programme for Catalunya 2014-2020.
European Commission's through the following R&D projects:
- H2020 I-BiDaaS project (Contract 780787)
- H2020 BioExcel Center of Excellence (Contracts 823830, and 675728)
- H2020 EuroHPC Joint Undertaking MEEP Project (Contract 946002)
- H2020 EuroHPC Joint Undertaking eFlows4HPC Project (Contract 955558)
- H2020 AI-Sprint project (Contract 101016577)
- H2020 PerMedCoE Center of Excellence (Contract 951773)
- Horizon Europe CAELESTIS project (Contract 101056886)
- Horizon Europe DT-Geo project (Contract 101058129)

License

Apache License Version 2.0, see LICENSE

Owner

Name: Workflows and Distributed Computing
Login: bsc-wdc
Kind: organization
Email: distributed_computing@bsc.es
Location: Barcelona

Website: https://www.bsc.es/discover-bsc/organisation/scientific-structure/workflows-and-distributed-computing
Repositories: 13
Profile: https://github.com/bsc-wdc

GitHub Events

Total

Watch event: 4
Delete event: 3
Member event: 1
Issue comment event: 6
Push event: 11
Pull request review comment event: 2
Pull request review event: 10
Pull request event: 11
Create event: 6

Last Year

Watch event: 4
Delete event: 3
Member event: 1
Issue comment event: 6
Push event: 11
Pull request review comment event: 2
Pull request review event: 10
Pull request event: 11
Create event: 6

Committers

Last synced: 9 months ago

All Time

Total Commits: 1,037
Total Committers: 21
Avg Commits per committer: 49.381
Development Distribution Score (DDS): 0.708

Past Year

Commits: 33
Committers: 1
Avg Commits per committer: 33.0
Development Distribution Score (DDS): 0.0

Top Committers

Name	Email	Commits
Javier Alvarez	j**z@b**s	303
FernandoVN98	f**8@g**m	202
Pol Alvarez Vecino	p**s@g**m	145
Salvi Solà Martinell	s**l@b**s	143
javicid	j****d	79
Guillem Casadesús Vila	g**a@g**m	49
Michal Choinski	m**i@b**s	28
Cristian Tatu	c**u@b**s	28
bscuser	b**r@l**n	17
compsuperscalar	c**r@g**m	9
Alex Barcelo	a**o@g**m	8
Raül Sirvent	R**t@b**s	6
fconejer	f**o@b**s	5
Nihad M	1**d@g**m	4
Sravya Garapati	s**i@i**m	4
Jorge Ejarque	j****e	2
DM126	3****6	1
nmammadl	n**i@b**s	1
Jorge Ejarque	j**e@b**s	1
Francesc Lordan	f**n@g**m	1
Nemanja Milosevic	n**m@g**m	1

Committer Domains (Top 20 + Academic)

bsc.es: 8 intuit.com: 1 localhost.localdomain: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 27
Total pull requests: 119
Average time to close issues: almost 2 years
Average time to close pull requests: 22 days
Total issue authors: 7
Total pull request authors: 10
Average comments per issue: 0.67
Average comments per pull request: 0.7
Merged pull requests: 79
Bot issues: 0
Bot pull requests: 4

Past Year

Issues: 0
Pull requests: 12
Average time to close issues: N/A
Average time to close pull requests: 7 days
Issue authors: 0
Pull request authors: 2
Average comments per issue: 0
Average comments per pull request: 0.67
Merged pull requests: 7
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

javicid (8)
cTatu (7)
salvisolamartinell (6)
kafkasl (2)
alexbarcelo (2)
aiq-kgielow (1)
vineel96 (1)

Pull Request Authors

FernandoVN98 (80)
cTatu (32)
michal-choinski (6)
dependabot[bot] (4)
compsuperscalar (3)
fjconejero (2)
alexbarcelo (2)
iraola (1)
jorgee (1)
tirkarthi (1)

Top Labels

Issue Labels

bug (11) enhancement (10) documentation (4) good first issue (3) easy (3) to solve in the future (2) algorithm related (2) python related (1) hard (1) discuss (1)

Pull Request Labels

dependencies (4)

Packages

Total packages: 1
Total downloads:
- pypi 145 last-month

Total dependent packages: 0
Total dependent repositories: 1
Total versions: 19
Total maintainers: 2

pypi.org: dislib

The distributed computing library on top of PyCOMPSs

Homepage: http://dislib.bsc.es
Documentation: http://dislib.bsc.es
License: Apache Software License
Latest release: 0.9.0
published over 2 years ago

Versions: 19
Dependent Packages: 0
Dependent Repositories: 1
Downloads: 145 Last month

Rankings

Forks count: 7.8%

Dependent packages count: 10.1%

Stargazers count: 10.2%

Average: 13.2%

Downloads: 16.1%

Dependent repos count: 21.5%

Maintainers (2)

COMPSs_Team jalvarez

Last synced: 6 months ago

Dependencies

requirements.txt pypi

cbor2 >=5.4.0
cvxpy >=1.1.5
numpy >=1.18.1,<=1.19.5
numpydoc >=0.8.0
scikit-learn >=0.22.1,<=0.24.1
scipy >=1.3.0

docs/source/requirements.txt pypi

cvxpy *
docutils <0.18
m2r *
mistune ==0.8.4
numpy ==1.16.0
numpydoc ==0.8.0
scikit-learn ==0.20.2
scipy ==1.2.0

setup.py pypi

cvxpy *
numpy *
scikit-learn *
scipy *

Dockerfile docker

bscwdc/dislib-base latest build

docker/Dockerfile docker

ubuntu 20.04 build

https://github.com/bsc-wdc/dislib

Science Score: 26.0%

Keywords

Keywords from Contributors

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

Distributed computing library implemented over PyCOMPSs programming model for HPC.

Introduction

Contents

Quickstart

Availability

Contributing

Citing dislib

Bibtex:

Acknowledgements

License

Owner

GitHub Events

Total

Last Year

Committers

All Time

Past Year

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: dislib

Rankings

Maintainers (2)

Dependencies