multiprocess
better multiprocessing and multithreading in Python
Science Score: 46.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
✓Committers with academic emails
1 of 9 committers (11.1%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (14.7%) to scientific vocabulary
Keywords from Contributors
Repository
better multiprocessing and multithreading in Python
Basic Info
- Host: GitHub
- Owner: uqfoundation
- License: other
- Language: Python
- Default Branch: master
- Homepage: http://multiprocess.rtfd.io
- Size: 1.66 MB
Statistics
- Stars: 676
- Watchers: 20
- Forks: 67
- Open Issues: 41
- Releases: 21
Metadata Files
README.md
multiprocess
better multiprocessing and multithreading in Python
About Multiprocess
multiprocess is a fork of multiprocessing. multiprocess extends multiprocessing to provide enhanced serialization, using dill. multiprocess leverages multiprocessing to support the spawning of processes using the API of the Python standard library's threading module. multiprocessing has been distributed as part of the standard library since Python 2.6.
multiprocess is part of pathos, a Python framework for heterogeneous computing.
multiprocess is in active development, so any user feedback, bug reports, comments,
or suggestions are highly appreciated. A list of issues is located at https://github.com/uqfoundation/multiprocess/issues, with a legacy list maintained at https://uqfoundation.github.io/project/pathos/query.
Major Features
multiprocess enables:
- objects to be transferred between processes using pipes or multi-producer/multi-consumer queues
- objects to be shared between processes using a server process or (for simple data) shared memory
multiprocess provides:
- equivalents of all the synchronization primitives in
threading - a
Poolclass to facilitate submitting tasks to worker processes - enhanced serialization, using
dill
Current Release
The latest released version of multiprocess is available from:
https://pypi.org/project/multiprocess
multiprocess is distributed under a 3-clause BSD license, and is a fork of multiprocessing.
Development Version
You can get the latest development version with all the shiny new features at: https://github.com/uqfoundation
If you have a new contribution, please submit a pull request.
Installation
multiprocess can be installed with pip::
$ pip install multiprocess
For Python 2, a C compiler is required to build the included extension module from source. Python 3 and binary installs do not require a C compiler.
Requirements
multiprocess requires:
python(orpypy), >=3.9setuptools, >=42dill, >=0.4.0
Basic Usage
The multiprocess.Process class follows the API of threading.Thread.
For example ::
from multiprocess import Process, Queue
def f(q):
q.put('hello world')
if __name__ == '__main__':
q = Queue()
p = Process(target=f, args=[q])
p.start()
print (q.get())
p.join()
Synchronization primitives like locks, semaphores and conditions are available, for example ::
>>> from multiprocess import Condition
>>> c = Condition()
>>> print (c)
<Condition(<RLock(None, 0)>), 0>
>>> c.acquire()
True
>>> print (c)
<Condition(<RLock(MainProcess, 1)>), 0>
One can also use a manager to create shared objects either in shared memory or in a server process, for example ::
>>> from multiprocess import Manager
>>> manager = Manager()
>>> l = manager.list(range(10))
>>> l.reverse()
>>> print (l)
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
>>> print (repr(l))
<Proxy[list] object at 0x00E1B3B0>
Tasks can be offloaded to a pool of worker processes in various ways, for example ::
>>> from multiprocess import Pool
>>> def f(x): return x*x
...
>>> p = Pool(4)
>>> result = p.map_async(f, range(10))
>>> print (result.get(timeout=1))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
When dill is installed, serialization is extended to most objects,
for example ::
>>> from multiprocess import Pool
>>> p = Pool(4)
>>> print (p.map(lambda x: (lambda y:y**2)(x) + x, xrange(10)))
[0, 2, 6, 12, 20, 30, 42, 56, 72, 90]
More Information
Probably the best way to get started is to look at the documentation at
http://multiprocess.rtfd.io. Also see multiprocess.tests for scripts that
demonstrate how multiprocess can be used to leverge multiple processes
to execute Python in parallel. You can run the test suite with
python -m multiprocess.tests. As multiprocess conforms to the
multiprocessing interface, the examples and documentation found at
http://docs.python.org/library/multiprocessing.html also apply to
multiprocess if one will import multiprocessing as multiprocess.
See https://github.com/uqfoundation/multiprocess/tree/master/py3.12/examples
for a set of examples that demonstrate some basic use cases and benchmarking
for running Python code in parallel. Please feel free to submit a ticket on
github, or ask a question on stackoverflow (@Mike McKerns). If you would
like to share how you use multiprocess in your work, please send an email
(to mmckerns at uqfoundation dot org).
Citation
If you use multiprocess to do research that leads to publication, we ask that you
acknowledge use of multiprocess by citing the following in your publication::
M.M. McKerns, L. Strand, T. Sullivan, A. Fang, M.A.G. Aivazis,
"Building a framework for predictive science", Proceedings of
the 10th Python in Science Conference, 2011;
http://arxiv.org/pdf/1202.1056
Michael McKerns and Michael Aivazis,
"pathos: a framework for heterogeneous computing", 2010- ;
https://uqfoundation.github.io/project/pathos
Please see https://uqfoundation.github.io/project/pathos or http://arxiv.org/pdf/1202.1056 for further information.
Owner
- Name: The UQ Foundation
- Login: uqfoundation
- Kind: organization
- Email: info@uqfoundation.org
- Location: Wilmington, DE
- Website: http://uqfoundation.org
- Repositories: 10
- Profile: https://github.com/uqfoundation
A nonprofit dedicated to the advancement of predictive science through research partnerships and education.
GitHub Events
Total
- Create event: 21
- Commit comment event: 2
- Release event: 1
- Issues event: 9
- Watch event: 50
- Delete event: 24
- Issue comment event: 16
- Push event: 43
- Pull request event: 47
- Fork event: 5
Last Year
- Create event: 21
- Commit comment event: 2
- Release event: 1
- Issues event: 9
- Watch event: 50
- Delete event: 24
- Issue comment event: 16
- Push event: 43
- Pull request event: 47
- Fork event: 5
Committers
Last synced: 8 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| mmckerns | m****s@c****u | 327 |
| dependabot[bot] | 4****] | 13 |
| Matthias Urlichs | m****s@u****e | 2 |
| Luke Pitt | L****t@g****m | 2 |
| Ԝеѕ | 5****r | 1 |
| Karolina Surma | 3****e | 1 |
| Hugo van Kemenade | h****k | 1 |
| Geoffrey Thomas | g****t@t****m | 1 |
| Roy Hvaara | h****a@g****m | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 118
- Total pull requests: 84
- Average time to close issues: 12 months
- Average time to close pull requests: 4 days
- Total issue authors: 73
- Total pull request authors: 12
- Average comments per issue: 2.73
- Average comments per pull request: 0.18
- Merged pull requests: 69
- Bot issues: 0
- Bot pull requests: 22
Past Year
- Issues: 9
- Pull requests: 46
- Average time to close issues: about 1 month
- Average time to close pull requests: 1 day
- Issue authors: 6
- Pull request authors: 4
- Average comments per issue: 1.11
- Average comments per pull request: 0.17
- Merged pull requests: 34
- Bot issues: 0
- Bot pull requests: 12
Top Authors
Issue Authors
- mmckerns (39)
- ughstudios (3)
- Nitrooo (2)
- Hana-Ali (2)
- jiasli (1)
- martinResearch (1)
- wfwzy2012 (1)
- yhcharles (1)
- lijinbio (1)
- Brave731 (1)
- ankostis (1)
- iritkatriel (1)
- noxdafox (1)
- jvavrek (1)
- juanmf (1)
Pull Request Authors
- mmckerns (64)
- dependabot[bot] (31)
- mgorny (2)
- befeleme (2)
- AmenRa (1)
- hugovk (1)
- geofft (1)
- dxdc (1)
- hvaara (1)
- claytonparnell (1)
- wesinator (1)
- smurfix (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 4
-
Total downloads:
- pypi 50,451,801 last-month
- Total docker downloads: 40,744,033
-
Total dependent packages: 336
(may contain duplicates) -
Total dependent repositories: 5,405
(may contain duplicates) - Total versions: 49
- Total maintainers: 2
pypi.org: multiprocess
better multiprocessing and multithreading in Python
- Homepage: https://github.com/uqfoundation/multiprocess
- Documentation: http://multiprocess.rtfd.io
- License: BSD-3-Clause
-
Latest release: 0.70.18
published 11 months ago
Rankings
Maintainers (1)
spack.io: py-multiprocess
Better multiprocessing and multithreading in Python
- Homepage: https://github.com/uqfoundation/multiprocess
- License: []
-
Latest release: 0.70.17
published 11 months ago
Rankings
Maintainers (1)
conda-forge.org: multiprocess
- Homepage: https://github.com/uqfoundation/multiprocess
- License: BSD-3-Clause
-
Latest release: 0.70.14
published over 3 years ago
Rankings
anaconda.org: multiprocess
`multiprocess` is a fork of `multiprocessing`. `multiprocess` extends `multiprocessing` to provide enhanced serialization, using `dill`. `multiprocess` leverages `multiprocessing` to support the spawning of processes using the API of the python standard library's `threading module. multiprocessing has been distributed as part of the standard library since python 2.6.
- Homepage: https://github.com/uqfoundation/multiprocess
- License: BSD-3-Clause
-
Latest release: 0.70.15
published over 2 years ago
Rankings
Dependencies
- alabaster ==0.7.12
- babel ==2.9.1
- docutils ==0.16
- imagesize ==1.3.0
- importlib-metadata ==4.11.3
- jinja2 ==3.1.1
- markupsafe ==2.1.1
- packaging ==21.3
- pygments ==2.11.2
- pyparsing ==3.0.8
- readthedocs-sphinx-search ==0.1.1
- requests ==2.27.1
- sphinx ==4.5.0
- sphinx-notfound-page ==0.8
- sphinx-rtd-theme ==1.0.0