Data

Tools

Indexes

Applications

Experiments

Open Source Science

rank-bm25

A Collection of BM25 Algorithms in Python

https://github.com/dorianbrown/rank_bm25

Science Score: 41.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: zenodo.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.9%) to scientific vocabulary

Keywords

algorithm bm25 information-retrieval ranking
Last synced: 6 months ago · JSON representation ·

Repository

A Collection of BM25 Algorithms in Python

Basic Info
  • Host: GitHub
  • Owner: dorianbrown
  • License: apache-2.0
  • Language: Python
  • Default Branch: master
  • Homepage:
  • Size: 43.9 KB
Statistics
  • Stars: 1,229
  • Watchers: 8
  • Forks: 97
  • Open Issues: 21
  • Releases: 2
Topics
algorithm bm25 information-retrieval ranking
Created about 7 years ago · Last pushed over 1 year ago
Metadata Files
Readme License Citation

README.md

Rank-BM25: A two line search engine

Build Status PyPI version PyPI - Downloads DOI PyPI - License

A collection of algorithms for querying a set of documents and returning the ones most relevant to the query. The most common use case for these algorithms is, as you might have guessed, to create search engines.

So far the algorithms that have been implemented are: - [x] Okapi BM25 - [x] BM25L - [x] BM25+ - [ ] BM25-Adpt - [ ] BM25T

These algorithms were taken from this paper, which gives a nice overview of each method, and also benchmarks them against each other. A nice inclusion is that they compare different kinds of preprocessing like stemming vs no-stemming, stopword removal or not, etc. Great read if you're new to the topic.

For those looking to use this in large scale production environments, I'd recommend you take a look at something like retriv, which is a much more performant python retrieval package. See #27

Installation

The easiest way to install this package is through pip, using bash pip install rank_bm25 If you want to be sure you're getting the newest version, you can install it directly from github with bash pip install git+ssh://git@github.com/dorianbrown/rank_bm25.git

Usage

For this example we'll be using the BM25Okapi algorithm, but the others are used in pretty much the same way.

Initalizing

First thing to do is create an instance of the BM25 class, which reads in a corpus of text and does some indexing on it: ```python from rank_bm25 import BM25Okapi

corpus = [ "Hello there good man!", "It is quite windy in London", "How is the weather today?" ]

tokenized_corpus = [doc.split(" ") for doc in corpus]

bm25 = BM25Okapi(tokenized_corpus)

``` Note that this package doesn't do any text preprocessing. If you want to do things like lowercasing, stopword removal, stemming, etc, you need to do it yourself.

The only requirements is that the class receives a list of lists of strings, which are the document tokens.

Ranking of documents

Now that we've created our document indexes, we can give it queries and see which documents are the most relevant: ```python query = "windy London" tokenized_query = query.split(" ")

docscores = bm25.getscores(tokenized_query)

array([0. , 0.93729472, 0. ])

``` Good to note that we also need to tokenize our query, and apply the same preprocessing steps we did to the documents in order to have an apples-to-apples comparison

Instead of getting the document scores, you can also just retrieve the best documents with ```python bm25.gettopn(tokenized_query, corpus, n=1)

['It is quite windy in London']

``` And that's pretty much it!

Owner

  • Name: Dorian Brown
  • Login: dorianbrown
  • Kind: user
  • Location: Amsterdam, NL
  • Company: Dorian Brown Analytics

Citation (CITATION) 

@software{rank_bm25,
  author = {Dorian Brown},
  title = {{Rank-BM25: A Collection of BM25 Algorithms in Python}},
  year = 2020,
  publisher = {Zenodo},
  doi = {10.5281/zenodo.4520057},
  url = {https://doi.org/10.5281/zenodo.4520057}
}

GitHub Events

Total
  • Issues event: 3
  • Watch event: 186
  • Issue comment event: 7
  • Pull request event: 1
  • Fork event: 10
Last Year
  • Issues event: 3
  • Watch event: 186
  • Issue comment event: 7
  • Pull request event: 1
  • Fork event: 10

Committers

Last synced: almost 3 years ago

All Time
  • Total Commits: 51
  • Total Committers: 8
  • Avg Commits per committer: 6.375
  • Development Distribution Score (DDS): 0.569
Top Committers
Name Email Commits
Dorian Brown d****n@g****m 22
Dorian Brown d****n@i****m 16
Dorian Brown m****l@d****v 8
Vít Novotný w****o@m****z 1
Dorian Brown b****c@a****t 1
Vít Novotný w****o@g****m 1
Sarthak Jain 5****6@u****m 1
nlp4whp n****p@g****m 1
Committer Domains (Top 20 + Academic)
apm3lc02xv9npjhd2.nl.europe.intranet: 1 mail.muni.cz: 1 dorianbrown.dev: 1 ing.com: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 33
  • Total pull requests: 16
  • Average time to close issues: about 1 year
  • Average time to close pull requests: 9 months
  • Total issue authors: 33
  • Total pull request authors: 13
  • Average comments per issue: 2.58
  • Average comments per pull request: 1.13
  • Merged pull requests: 6
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 2
  • Pull requests: 3
  • Average time to close issues: 2 minutes
  • Average time to close pull requests: N/A
  • Issue authors: 2
  • Pull request authors: 3
  • Average comments per issue: 0.0
  • Average comments per pull request: 0.33
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • Smu-Tan (1)
  • Witiko (1)
  • nkarahan-ing (1)
  • fortyfourforty (1)
  • Grecil (1)
  • soumya-ranjan-sahoo (1)
  • lavarthan (1)
  • bohnpessatti (1)
  • lambdaofgod (1)
  • karndeepsingh (1)
  • RakshaAg (1)
  • Alkacid (1)
  • ghost (1)
  • ramsey-coding (1)
  • kripper (1)
Pull Request Authors
  • Witiko (4)
  • dorianbrown (3)
  • xujiang1 (2)
  • danerlt (2)
  • jankovicsandras (2)
  • raoashish10 (1)
  • mariusjohan (1)
  • chenrulongmaster (1)
  • LowinLi (1)
  • nlp4whp (1)
  • Aashu-Adhikari (1)
  • Sarthakjain1206 (1)
Top Labels
Issue Labels
enhancement (4) bug (1)
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 1,730,844 last-month
  • Total docker downloads: 991
  • Total dependent packages: 69
  • Total dependent repositories: 345
  • Total versions: 4
  • Total maintainers: 1
pypi.org: rank-bm25

Various BM25 algorithms for document ranking

  • Versions: 4
  • Dependent Packages: 69
  • Dependent Repositories: 345
  • Downloads: 1,730,844 Last month
  • Docker Downloads: 991
Rankings
Dependent packages count: 0.3%
Dependent repos count: 0.8%
Downloads: 0.8%
Average: 1.9%
Stargazers count: 2.3%
Docker downloads count: 2.6%
Forks count: 4.5%
Maintainers (1)
dorianbrown
Last synced: 6 months ago

Dependencies

requirements.txt pypi
  • numpy *
setup.py pypi
  • numpy *
.github/workflows/pytest.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
.github/workflows/python-publish.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite