loky

Robust and reusable Executor for joblib

https://github.com/joblib/loky

Science Score: 64.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
✓
Academic publication links
Links to: zenodo.org
✓
Committers with academic emails
1 of 20 committers (5.0%) from academic institutions
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (16.8%) to scientific vocabulary

Keywords

multiprocessing-library python

Keywords from Contributors

caching memoization multiprocessing parallel-computing threading closember version

Last synced: 6 months ago · JSON representation ·

Repository

Robust and reusable Executor for joblib

Basic Info

Host: GitHub
Owner: joblib
License: bsd-3-clause
Language: Python
Default Branch: master
Homepage: http://loky.readthedocs.io/en/stable/
Size: 1.33 MB

Statistics

Stars: 581
Watchers: 14
Forks: 48
Open Issues: 53
Releases: 0

Topics

multiprocessing-library python

Created about 10 years ago · Last pushed 6 months ago

Metadata Files

Readme Changelog License Citation

Reusable Process Pool Executor

Goal

The aim of this project is to provide a robust, cross-platform and cross-version implementation of the ProcessPoolExecutor class of concurrent.futures. It notably features:

Consistent and robust spawn behavior: All processes are started using fork + exec on POSIX systems. This ensures safer interactions with third party libraries. On the contrary, multiprocessing.Pool uses fork without exec by default, causing third party runtimes to crash (e.g. OpenMP, macOS Accelerate...).
Reusable executor: strategy to avoid re-spawning a complete executor every time. A singleton executor instance can be reused (and dynamically resized if necessary) across consecutive calls to limit spawning and shutdown overhead. The worker processes can be shutdown automatically after a configurable idling timeout to free system resources.
Transparent cloudpickle integration: to call interactively defined functions and lambda expressions in parallel. It is also possible to register a custom pickler implementation to handle inter-process communications.
No need for ``if __name == "main":in scripts__: thanks to the use ofcloudpickleto call functions defined in themain`` module, it is not required to protect the code calling parallel functions under Windows.
Deadlock free implementation: one of the major concern in standard multiprocessing and concurrent.futures modules is the ability of the Pool/Executor to handle crashes of worker processes. This library intends to fix those possible deadlocks and send back meaningful errors. Note that the implementation of concurrent.futures.ProcessPoolExecutor that comes with Python 3.7+ is as robust as the executor from loky but the latter also works for older versions of Python.

Installation

The recommended way to install loky is with pip, bash pip install loky

loky can also be installed from sources using bash git clone https://github.com/joblib/loky cd loky python setup.py install

Note that loky has an optional dependency on psutil to allow early memory leak detections.

Usage

The basic usage of loky relies on the get_reusable_executor, which internally manages a custom ProcessPoolExecutor object, which is reused or re-spawned depending on the context.

```python import os from time import sleep from loky import getreusableexecutor

def say_hello(k): pid = os.getpid() print(f"Hello from {pid} with arg {k}") sleep(.01) return pid

Create an executor with 4 worker processes that will

automatically shutdown after idling for 2s

executor = getreusableexecutor(max_workers=4, timeout=2)

res = executor.submit(say_hello, 1) print("Got results:", res.result())

results = executor.map(sayhello, range(50)) nworkers = len(set(results)) print("Number of used processes:", nworkers) assert nworkers == 4 ```

For more advanced usage, see our documentation

Workflow to contribute

To contribute to loky, first create an account on github. Once this is done, fork the loky repository to have your own repository, clone it using 'git clone' on the computers where you want to work. Make your changes in your clone, push them to your github account, test them on several computers, and when you are happy with them, send a pull request to the main repository.

Running the test suite

To run the test suite, you need the pytest (version >= 3) and psutil modules. From the root of the project, run the test suite using:

sh pip install -e . pytest .

Why was the project named `loky`?

While developing loky, we had some bad experiences trying to debug deadlocks when using multiprocessing.Pool and concurrent.futures.ProcessPoolExecutor, especially when calling functions with non-picklable arguments or returned values at the beginning of the project. When we had to choose a name, we had dealt with so many deadlocks that we wanted some kind of invocation to repel them! Hence loky: a mix of a god, locks and the y that make it somehow cooler and nicer : (and also less likely to result in name conflict in google results ^^).

Fixes to avoid those deadlocks in concurrent.futures were also contributed upstream in Python 3.7+, as a less mystical way to repel the deadlocks :D

Acknowledgement

This work is supported by the Center for Data Science, funded by the IDEX Paris-Saclay, ANR-11-IDEX-0003-02

Owner

Name: joblib
Login: joblib
Kind: organization

Repositories: 8
Profile: https://github.com/joblib

Citation (CITATION.cff)

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "Moreau"
  given-names: "Thomas"
  orcid: "https://orcid.org/0000-0002-1523-3419"
- family-names: "Glaser"
  given-names: "Pierre"
  orcid: "https://orcid.org/0000-0003-4146-973X"
- family-names: "Yurchak"
  given-names: "Roman"
  orcid: "https://orcid.org/0000-0002-2565-4444"
- family-names: "Grisel"
  given-names: "Olivier"
  orcid: "https://orcid.org/0000-0002-8226-4012"
title: "Loky"
version: latest
doi: 10.5281/zenodo.5770406
date-released: 2017-06-20
url: "https://github.com/joblib/loky"

GitHub Events

Total

Create event: 7
Release event: 6
Issues event: 17
Watch event: 47
Delete event: 2
Issue comment event: 100
Push event: 35
Pull request review comment event: 44
Pull request review event: 73
Pull request event: 59
Fork event: 3

Last Year

Create event: 7
Release event: 6
Issues event: 17
Watch event: 47
Delete event: 2
Issue comment event: 100
Push event: 35
Pull request review comment event: 44
Pull request review event: 73
Pull request event: 59
Fork event: 3

Committers

Last synced: 11 months ago

All Time

Total Commits: 677
Total Committers: 20
Avg Commits per committer: 33.85
Development Distribution Score (DDS): 0.508

Past Year

Commits: 24
Committers: 4
Avg Commits per committer: 6.0
Development Distribution Score (DDS): 0.583

Top Committers

Name	Email	Commits
Thomas Moreau	t**l@g**m	333
Olivier Grisel	o**l@e**g	251
Pierre Glaser	p**r@m**m	36
Jérémie du Boisberranger	3****b	30
Guillaume Lemaitre	g**8@g**m	6
Loïc Estève	l**e@y**m	3
Roman Yurchak	r**k@g**m	3
Haim Daniel	h****n	2
Michał Górny	m**y@g**g	2
Albert Thomas	a**8@g**m	1
Alexandre Abadie	a**e@i**r	1
Bas Nijholt	b**t@g**m	1
Christopher J. Wright	c**h@g**m	1
Joan Massich	m**k@g**m	1
Lukasz Migas	l**s@y**m	1
Moin Nadeem	M**m@m**m	1
PattyC	2****p	1
chkoar	c****r	1
lilinjie	1****e	1
pmav99	p****9	1

Committer Domains (Top 20 + Academic)

moinnadeem.com: 1 inria.fr: 1 gentoo.org: 1 ymail.com: 1 msn.com: 1 ensta.org: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 50
Total pull requests: 148
Average time to close issues: 4 months
Average time to close pull requests: about 1 month
Total issue authors: 28
Total pull request authors: 19
Average comments per issue: 2.38
Average comments per pull request: 2.04
Merged pull requests: 112
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 10
Pull requests: 64
Average time to close issues: 4 days
Average time to close pull requests: 5 days
Issue authors: 6
Pull request authors: 6
Average comments per issue: 1.6
Average comments per pull request: 1.8
Merged pull requests: 49
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

ogrisel (18)
tomMoral (4)
lesteve (2)
sjdv1982 (2)
arquolo (1)
Barry1 (1)
DamianBarabonkovQC (1)
pmav99 (1)
louisabraham (1)
Sikerdebaard (1)
josephschorr (1)
khoben (1)
ryandvmartin (1)
dansleboby (1)
glemaitre (1)

Pull Request Authors

ogrisel (46)
jeremiedbb (44)
tomMoral (28)
glemaitre (7)
lesteve (6)
ChrisLoveringSendient (2)
haim0n (2)
alphabetagamer (2)
skreuzer (2)
schlerp (1)
bang78945 (1)
uniontech-lilinjie (1)
arquolo (1)
pmav99 (1)
hoodmane (1)

Top Labels

Issue Labels

bug (4) help wanted (1) regression (1) free-threading (1)

Pull Request Labels

bug (2) testing (1) free-threading (1)

Packages

Total packages: 3
Total downloads:
- pypi 324,217 last-month
Total docker downloads: 558

Total dependent packages: 39
(may contain duplicates)
Total dependent repositories: 144
(may contain duplicates)
Total versions: 65
Total maintainers: 4

pypi.org: loky

A robust implementation of concurrent.futures.ProcessPoolExecutor

Homepage: http://loky.readthedocs.io
Documentation: https://loky.readthedocs.io/
License: BSD 3-Clause License Copyright (c) 2017, Olivier Grisel & Thomas Moreau. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Latest release: 3.5.6
published 6 months ago

Versions: 45
Dependent Packages: 34
Dependent Repositories: 132
Downloads: 324,217 Last month
Docker Downloads: 558

Rankings

Dependent packages count: 0.4%

Downloads: 0.6%

Dependent repos count: 1.3%

Average: 2.3%

Docker downloads count: 2.3%

Stargazers count: 2.9%

Forks count: 6.1%

Maintainers (4)

tommoral ogrisel jeremiedbb pierreglaser

Last synced: 6 months ago

conda-forge.org: loky

Homepage: https://github.com/joblib/loky
License: BSD-3-Clause
Latest release: 3.3.0
published over 3 years ago

Versions: 19
Dependent Packages: 5
Dependent Repositories: 6

Rankings

Dependent packages count: 10.4%

Dependent repos count: 13.9%

Average: 17.3%

Stargazers count: 18.4%

Forks count: 26.6%

Last synced: 6 months ago

anaconda.org: loky

A robust, cross-platform and cross-version implementation of concurrent.futures.ProcessPoolExecutor with many features.

Homepage: https://github.com/joblib/loky
License: BSD-3-Clause
Latest release: 3.4.0
published almost 3 years ago

Versions: 1
Dependent Packages: 0
Dependent Repositories: 6

Rankings

Stargazers count: 31.0%

Forks count: 38.7%

Average: 40.7%

Dependent repos count: 42.0%

Dependent packages count: 51.0%

Last synced: 6 months ago

Dependencies

dev-requirements.txt pypi

coverage * development
cython * development
numpy * development
psutil * development
pytest * development
pytest-coverage * development
pytest-timeout * development

docs/requirements.txt pypi

matplotlib *
sphinx-gallery *

setup.py pypi

cloudpickle *

pyproject.toml pypi

tests/_openmp/setup.py pypi

loky

Science Score: 64.0%

Keywords

Keywords from Contributors

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

Reusable Process Pool Executor

Goal

Installation

Usage

Create an executor with 4 worker processes that will

automatically shutdown after idling for 2s

Workflow to contribute

Running the test suite

Why was the project named loky?

Acknowledgement

Owner

Citation (CITATION.cff)

GitHub Events

Total

Last Year

Committers

All Time

Past Year

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: loky

Rankings

Maintainers (4)

conda-forge.org: loky

Rankings

anaconda.org: loky

Rankings

Dependencies

Why was the project named `loky`?