Data

Tools

Indexes

Applications

Experiments

Open Source Science

pipeline-turbo

Accelerate your processing pipeline

https://github.com/dreji18/pipeline-turbo

Science Score: 44.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.8%) to scientific vocabulary
Last synced: 7 months ago · JSON representation ·

Repository

Accelerate your processing pipeline

Basic Info
  • Host: GitHub
  • Owner: dreji18
  • License: mit
  • Language: Jupyter Notebook
  • Default Branch: main
  • Size: 118 KB
Statistics
  • Stars: 7
  • Watchers: 1
  • Forks: 1
  • Open Issues: 0
  • Releases: 0
Created over 3 years ago · Last pushed over 3 years ago
Metadata Files
Readme License Citation

README.md

Pipeline-turbo

pipeline-turbo is a package that will accelerate your processing pipeline. It works with the multi-threading concept in the background. It has been successful in both CPU and GPU tasks.

Downloads CI CI

The only pre-requisite is to load the function running for a single process and adjust the threads according to your resource availability. turbo

Read more about threading here: https://www.activestate.com/blog/how-to-manage-threads-in-python/

Installation

Use the package manager pip to install pipeline-turbo

bash pip install pipeline-turbo

Example Usage

```python

let's get some data for processing

sentences = ["Nevertheless, Trump and other Republicans have tarred the protests as havens for terrorists intent on destroying property.", "Billie Eilish issues apology for mouthing an anti-Asian derogatory term in a resurfaced video.", "Christians should make clear that the perpetuation of objectionable vaccines and the lack of alternatives is a kind of coercion.", "There have been a protest by a group of people", "While emphasizing he’s not singling out either party, Cohen warned about the danger of normalizing white supremacist ideology."]

sentences = sentences * 100

Create your process - Here is an example of running a bias detection model across few sentences

from transformers import AutoTokenizer, TFAutoModelForSequenceClassification from transformers import pipeline tokenizer = AutoTokenizer.frompretrained("d4data/bias-detection-model") model = TFAutoModelForSequenceClassification.frompretrained("d4data/bias-detection-model") classifier = pipeline('text-classification', model=model, tokenizer=tokenizer)

def biasclassification(text): out = classifier(text) #label = out[0]['label'] #probability_ = out[0]['score']

return out

without turbo, looping across all the sentences - Normal Method

outlist = [] for sent in sentences: out = biasclassification(sent) out_list.append(out)

with turbo, call the turbo_threading function

""" 1. Each of the item in 'sentences' list has to be iterated and that has to be defined as the first argument 2. It should be followed by the function and its other arguments (if there are additional arguments for the function) 3. Define the thread based on your resource availability (5, 10 would be ideal based on your resources) """ from pipelineturbo.turbo import turbothreading # import the turbo threading function which does the magic turboout = turbothreading(sentences,biasclassification, numthreads=5)

""" Note: You can pass any number of arguments inside the function, but the iterable list has to be defined first The performance varies based on the processing speed of your machine/compute """ ```

About

This package is created by Deepak John Reji, Afreen Aman. It was first used to speed up some deep learning pipeline projects and later made it open source. This can be used for normal CPU process as well.

License

MIT License

Owner

  • Name: Deepak John Reji
  • Login: dreji18
  • Kind: user
  • Location: Bangalore
  • Company: ERM

I am an NLP practitioner with experience in developing and structuring solutions for the data science environment

Citation (CITATION.cff) 

@software{pipeline-turbo,
  author = {Deepak John Reji, Afreen Aman},
  title = {pipeline-turbo},
  url = {https://github.com/dreji18/pipeline-turbo},
  version = {0.0.5},
  year = {2022}
}

GitHub Events

Total
Last Year

Committers

Last synced: about 3 years ago

All Time
  • Total Commits: 18
  • Total Committers: 2
  • Avg Commits per committer: 9.0
  • Development Distribution Score (DDS): 0.278
Top Committers
Name Email Commits
Deepak John Reji 4****8@u****m 13
Afreen Aman 6****n@u****m 5

Issues and Pull Requests

Last synced: 8 months ago

All Time
  • Total issues: 1
  • Total pull requests: 0
  • Average time to close issues: about 2 hours
  • Average time to close pull requests: N/A
  • Total issue authors: 1
  • Total pull request authors: 0
  • Average comments per issue: 2.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • amrrs (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 18 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 0
  • Total versions: 6
  • Total maintainers: 2
pypi.org: pipeline-turbo

Accelerate your processing pipeline

  • Versions: 6
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 18 Last month
Rankings
Dependent packages count: 6.6%
Average: 26.7%
Forks count: 30.5%
Dependent repos count: 30.6%
Stargazers count: 39.1%
Maintainers (2)
dreji18 Afreen12
Last synced: 8 months ago

Dependencies

requirements.txt pypi
  • tqdm *