Data

Tools

Indexes

Applications

Experiments

Open Source Science

pypette

Ridiculously simple flow controller for building complex pipelines

https://github.com/csurfer/pypette

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (15.0%) to scientific vocabulary

Keywords

multiprocessing multithreading python threads

Keywords from Contributors

interactive serializer cycles packaging network-simulation shellcodes hacking autograding observability genomics
Last synced: 6 months ago · JSON representation

Repository

Ridiculously simple flow controller for building complex pipelines

Basic Info
Statistics
  • Stars: 264
  • Watchers: 9
  • Forks: 9
  • Open Issues: 2
  • Releases: 0
Topics
multiprocessing multithreading python threads
Created over 8 years ago · Last pushed 10 months ago
Metadata Files
Readme Changelog License

README.md

pypiv pyv Build Status Coverage Status License

pypette (to be read as pipette) is a module which makes building pipelines ridiculously simple, allowing users to control the flow with minimal instructions.

Features

  • Ridiculously simple interface.
  • Ability to view pipeline structure within the comfort of a terminal.
  • Run pipeline in exception resilient way if needed.
  • Create dependencies on pipelines easily.
  • Generate a easy to view/understand report within the comfort of a terminal.

Setup

Using pip

bash pip install pypette

Directly from the repository

bash git clone https://github.com/csurfer/pypette.git python pypette/setup.py install

Documentation

Detailed documentation can be found at https://csurfer.github.io/pypette

Structures

Job

The basic unit of execution, say a python method or a callable.

```python from pypette import Job

def print_hello(): print("Hello!")

def printhellomsg(msg): print("Hello " + msg + "!")

Job without arguments

j1 = Job(print_hello)

Job with arguments specified as argument list

j2 = Job(printhellomsg, args=("pypette is simple",))

Job with arguments specified as key word arguments

j3 = Job(printhellomsg, kwargs={"msg":"pypette is simple"}) ```

BashJob

The basic unit of execution, which runs a bash command.

```python from pypette import BashJob

Job with bash commands

b1 = BashJob(['ls', '-l']) b2 = BashJob(['pwd']) ```

Pipe

Structure to specify the flow in which the jobs need to be executed. The whole interface consists of only 4 methods.

```python from pypette import Pipe

1. Create a new Pipe

p = Pipe('TestPipe')

2. Add jobs to execute. (Assuming job_list is a list of python/bash jobs)

To run the jobs in job_list in order one after the other where each job waits

for the job before it to finish.

p.addjobs(joblist)

To run the jobs in job_list parallelly and run the next step only after all

jobs in job list finish.

p.addjobs(joblist, runinparallel=True)

Add jobs in a builder format.

p.addstage(job1).addstage(job2) # To add jobs in series. p.add_stage(job1, job2) # To add jobs in parallel. ```

Building complex pipelines

Jobs submitted to pipeline should be callables i.e. structures which can be run. This means python methods, lambdas etc qualify.

What about Pipe itself?

Of course, it is a callable and you can submit a pipe object to be run along with regular jobs. This way you can build small pipelines which achieve a specific task and then combine them to create more complex pipelines.

```python from pypette import BashJob, Job, Pipe

def welcome(): print("Welcome user!")

def havefun(): print("Have fun!")

def goodbye(): print("Goodbye!")

Build a simple pipeline

p1 = Pipe('Fun') p1.add_jobs([ Job(havefun), ])

Include simple pipeline into a complicated pipeline

p2 = Pipe('Overall') p2.add_jobs([ Job(welcome), p1, Job(goodbye), BashJob(['ls', '-l']), BashJob(['pwd']) ])

p2.run() # This first runs welcome, then runs p1 pipeline then runs goodbye. ```

Example pipeline

An example pipeline and its code are included in examples folder.

Visualizing the pipeline using graph()

Pipeline objects have a method called graph() which helps visualize the pipeline within the comfort of your terminal. The graph is recursive in nature and it visualizes everything that will be run if we call run() on the pipe object.

Visualizing the top-level pipeline in examples/basic.py led to the following visualization.

Running the entire pipeline.

The only thing you need to do at this point to run the entire pipeline is to call run() on your pipeline object.

Reporting the entire pipeline.

The only thing you need to do at this point to get the report of entire pipeline is to call report() on your pipeline object.

Contributing

Bug Reports and Feature Requests

Please use issue tracker for reporting bugs or feature requests.

Development

  1. Checkout the repository.
  2. Make your changes and add/update relavent tests.
  3. Install poetry using pip install poetry.
  4. Run poetry install to create project's virtual environment.
  5. Run tests using poetry run tox (Any python versions which you don't have checked out will fail this). Fix failing tests and repeat.
  6. Make documentation changes that are relavant.
  7. Install pre-commit using pip install pre-commit and run pre-commit run --all-files to do lint checks.
  8. Generate documentation using poetry run sphinx-build -b html docs/ docs/_build/html.
  9. Generate requirements.txt for automated testing using poetry export --dev --without-hashes -f requirements.txt > requirements.txt.
  10. Commit the changes and raise a pull request.

Buy the developer a cup of coffee!

If you found the utility helpful you can buy me a cup of coffee using

Donate

Owner

  • Name: Vishwas B Sharma
  • Login: csurfer
  • Kind: user
  • Location: Foster City
  • Company: Lyft

Philomath

GitHub Events

Total
  • Watch event: 1
  • Push event: 1
  • Fork event: 1
  • Create event: 1
Last Year
  • Watch event: 1
  • Push event: 1
  • Fork event: 1
  • Create event: 1

Committers

Last synced: 9 months ago

All Time
  • Total Commits: 105
  • Total Committers: 2
  • Avg Commits per committer: 52.5
  • Development Distribution Score (DDS): 0.01
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Vishwas B Sharma s****8@g****m 104
dependabot[bot] 4****] 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 5
  • Total pull requests: 16
  • Average time to close issues: 12 months
  • Average time to close pull requests: about 1 month
  • Total issue authors: 4
  • Total pull request authors: 5
  • Average comments per issue: 3.0
  • Average comments per pull request: 0.81
  • Merged pull requests: 7
  • Bot issues: 0
  • Bot pull requests: 7
Past Year
  • Issues: 0
  • Pull requests: 1
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 1
View more stats
Top Authors
Issue Authors
  • kstrauser (2)
  • MintCollie (1)
  • jfunez (1)
  • liuliqiang (1)
Pull Request Authors
  • csurfer (6)
  • dependabot[bot] (6)
  • johnlunney (2)
  • codeflash-ai[bot] (1)
  • stsievert (1)
Top Labels
Issue Labels
enhancement (2) invalid (1)
Pull Request Labels
dependencies (6) ⚡️ codeflash (1)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 1,587 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 12
  • Total maintainers: 1
pypi.org: pypette

pypette (to be read as pipette) is a module which makes building pipelines ridiculously simple, allowing users to control the flow with minimal instructions.

  • Versions: 12
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 1,587 Last month
Rankings
Stargazers count: 3.9%
Dependent packages count: 10.0%
Forks count: 10.9%
Average: 12.0%
Downloads: 13.7%
Dependent repos count: 21.7%
Maintainers (1)
csurfer
Last synced: 6 months ago

Dependencies

poetry.lock pypi
  • atomicwrites 1.4.0 develop
  • attrs 21.2.0 develop
  • backports.entry-points-selectable 1.1.0 develop
  • distlib 0.3.2 develop
  • filelock 3.0.12 develop
  • importlib-metadata 4.8.1 develop
  • importlib-resources 5.2.2 develop
  • iniconfig 1.1.1 develop
  • platformdirs 2.3.0 develop
  • pluggy 1.0.0 develop
  • py 1.10.0 develop
  • pytest 6.2.5 develop
  • toml 0.10.2 develop
  • tox 3.24.3 develop
  • typing-extensions 3.10.0.2 develop
  • virtualenv 20.7.2 develop
  • zipp 3.5.0 develop
  • alabaster 0.7.12
  • babel 2.9.1
  • certifi 2021.5.30
  • charset-normalizer 2.0.4
  • colorama 0.4.4
  • crayons 0.4.0
  • docutils 0.17.1
  • idna 3.2
  • imagesize 1.2.0
  • jinja2 3.0.1
  • livereload 2.6.3
  • markupsafe 2.0.1
  • packaging 21.0
  • pygments 2.10.0
  • pyparsing 2.4.7
  • pytz 2021.1
  • requests 2.26.0
  • six 1.16.0
  • snowballstemmer 2.1.0
  • sphinx 4.2.0
  • sphinx-autobuild 2021.3.14
  • sphinxcontrib-applehelp 1.0.2
  • sphinxcontrib-devhelp 1.0.2
  • sphinxcontrib-htmlhelp 2.0.0
  • sphinxcontrib-jsmath 1.0.1
  • sphinxcontrib-qthelp 1.0.3
  • sphinxcontrib-serializinghtml 1.1.5
  • tornado 6.1
  • urllib3 1.26.6
pyproject.toml pypi
  • pytest ^6.2.5 develop
  • tox ^3.24.3 develop
  • Sphinx ^4.1.2
  • crayons ^0.4.0
  • python ^3.6
  • sphinx-autobuild ^2021.3.14
requirements.txt pypi
  • alabaster ==0.7.12
  • atomicwrites ==1.4.0
  • attrs ==21.2.0
  • babel ==2.9.1
  • backports.entry-points-selectable ==1.1.0
  • certifi ==2021.5.30
  • charset-normalizer ==2.0.4
  • colorama ==0.4.4
  • crayons ==0.4.0
  • distlib ==0.3.2
  • docutils ==0.17.1
  • filelock ==3.0.12
  • idna ==3.2
  • imagesize ==1.2.0
  • importlib-metadata ==4.8.1
  • importlib-resources ==5.2.2
  • iniconfig ==1.1.1
  • jinja2 ==3.0.1
  • livereload ==2.6.3
  • markupsafe ==2.0.1
  • packaging ==21.0
  • platformdirs ==2.3.0
  • pluggy ==1.0.0
  • py ==1.10.0
  • pygments ==2.10.0
  • pyparsing ==2.4.7
  • pytest ==6.2.5
  • pytz ==2021.1
  • requests ==2.26.0
  • six ==1.16.0
  • snowballstemmer ==2.1.0
  • sphinx ==4.2.0
  • sphinx-autobuild ==2021.3.14
  • sphinxcontrib-applehelp ==1.0.2
  • sphinxcontrib-devhelp ==1.0.2
  • sphinxcontrib-htmlhelp ==2.0.0
  • sphinxcontrib-jsmath ==1.0.1
  • sphinxcontrib-qthelp ==1.0.3
  • sphinxcontrib-serializinghtml ==1.1.5
  • toml ==0.10.2
  • tornado ==6.1
  • tox ==3.24.3
  • typing-extensions ==3.10.0.2
  • urllib3 ==1.26.6
  • virtualenv ==20.7.2
  • zipp ==3.5.0
setup.py pypi
  • crayons *
.github/workflows/pytest.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
  • codecov/codecov-action v2 composite