Data

Tools

Indexes

Applications

Experiments

Open Source Science

match

Match tokenized words and phrases within the original, untokenized, often messy, text.

https://github.com/educationaltestingservice/match

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (9.6%) to scientific vocabulary
Last synced: 11 months ago · JSON representation

Repository

Match tokenized words and phrases within the original, untokenized, often messy, text.

Basic Info
  • Host: GitHub
  • Owner: EducationalTestingService
  • License: apache-2.0
  • Language: Python
  • Default Branch: develop
  • Size: 244 KB
Statistics
  • Stars: 19
  • Watchers: 19
  • Forks: 5
  • Open Issues: 3
  • Releases: 2
Created over 12 years ago · Last pushed over 3 years ago
Metadata Files
Readme License

README.rst 

match
=====

|Build Status|

|Latest Conda Version|\ |Latest PyPI Version|\ |Python Versions|

The purpose of the module ``Match`` is to get the offsets (as well as
the string between those offsets, for debugging) of a cleaned-up,
tokenized string from its original, untokenized source. “Big deal,” you
might say, but this is actually a pretty difficult task if the original
text is sufficiently messy, not to mention rife with Unicode characters.

Consider some text, stored in a variable ``original_text``, like:

::

   I   am writing a letter !  Sometimes,I forget to put spaces (and do weird stuff with punctuation)  ?  J'aurai une pomme, s'il vous plâit !

This will/should/might be properly tokenized as:

.. code:: python

   [['I', 'am', 'writing', 'a', 'letter', '!'],
    ['Sometimes', ',', 'I', 'forget', 'to', 'put', 'spaces', '-LRB-', 'and', 'do', 'weird', 'stuff', 'with', 'punctuation', '-RRB-', '?'],
    ["J'aurai", 'une', 'pomme', ',', "s'il", 'vous', 'plâit', '!']]

Now:

.. code:: python

   In [2]: import match

   In [3]: match.match(original_text, ['-LRB-', 'and', 'do', 'weird', 'stuff', 'with', 'punctuation', '-RRB-'])
   Out[3]: [(60, 97, '(and do weird stuff with punctuation)')]

   In [4]: match.match(original_text, ['I', 'am', 'writing', 'a', 'letter', '!'])
   Out[4]: [(0, 25, 'I   am writing a letter !')]

   In [5]: match.match(original_text, ["s'il", 'vous', 'plâit', '!'])
   Out[5]: [(121, 138, "s'il vous plâit !")]

The return type from ``match()`` is a ``list`` because it will return
*all* occurrences of the argument, be it a ``list`` of tokens or a
single ``string`` (word):

.. code:: python

   In [6]: match.match(original_text, "I")
   Out[6]: [(0, 1, 'I'), (37, 38, 'I')]

When passing in a single ``string``, ``match()`` is expecting that
``string`` to be a single word or token. Thus:

.. code:: python

   In [7]: match.match("****because,the****", "because , the")
   Out[7]: []

Try passing in ``"because , the".split(' ')`` instead, or better yet,
the output from a proper tokenizer.

For convenience, a function called ``match_lines()`` is provided:

.. code:: python

   In [8]: match.match_lines(original_text, [
      ...: ['-LRB-', 'and', 'do', 'weird', 'stuff', 'with', 'punctuation', '-RRB-'],
      ...: ['I', 'am', 'writing', 'a', 'letter', '!'],
      ...: "I"
      ...: ])
   Out[8]:
   [(0, 1, 'I'),
    (0, 25, 'I   am writing a letter !'),
    (37, 38, 'I'),
    (60, 97, '(and do weird stuff with punctuation)')]

The values returned will always be sorted by their offsets.

Installation
------------

``pip install match`` or ``conda install -c ets match``

Requirements
------------

-  Python >= 3.8
-  `nltk `__
-  `regex `__

Documentation
-------------

`Here! `__.

.. |Build Status| image:: https://github.com/EducationalTestingService/match/actions/workflows/python-test.yml/badge.svg
   :target: https://github.com/EducationalTestingService/match/actions/workflows/python-test.yml/
.. |Latest Conda Version| image:: https://img.shields.io/conda/v/ets/match
   :target: https://anaconda.org/ets/match
.. |Latest PyPI Version| image:: https://img.shields.io/pypi/v/match
   :target: https://pypi.org/project/match/
.. |Python Versions| image:: https://img.shields.io/pypi/pyversions/match
   :target: https://pypi.python.org/pypi/match/

Owner

  • Name: ETS
  • Login: EducationalTestingService
  • Kind: organization
  • Email: opensource@ets.org
  • Location: Princeton, NJ

Educational Testing Service

GitHub Events

Total
Last Year

Committers

Last synced: almost 3 years ago

All Time
  • Total Commits: 101
  • Total Committers: 8
  • Avg Commits per committer: 12.625
  • Development Distribution Score (DDS): 0.594
Past Year
  • Commits: 1
  • Committers: 1
  • Avg Commits per committer: 1.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Napolitano, Diane d****o@e****g 41
Diane d****o@g****m 39
tamarl08 1****8 12
Nitin Madnani n****i@e****g 4
Lars Buitinck l****k@e****l 2
Michael Overmeyer m****r@y****a 1
briverse17 5****7 1
Vyacheslav Andreyev s****v@e****g 1
Committer Domains (Top 20 + Academic)
ets.org: 3 yahoo.ca: 1 esciencecenter.nl: 1

Issues and Pull Requests

Last synced: 12 months ago

All Time
  • Total issues: 10
  • Total pull requests: 13
  • Average time to close issues: 8 months
  • Average time to close pull requests: about 1 month
  • Total issue authors: 7
  • Total pull request authors: 8
  • Average comments per issue: 1.1
  • Average comments per pull request: 1.69
  • Merged pull requests: 10
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • dmnapolitano (4)
  • tpeng (1)
  • larsmans (1)
  • whardier (1)
  • geekwolf (1)
  • amadanmath (1)
Pull Request Authors
  • dmnapolitano (3)
  • tamarl08 (3)
  • slava92 (2)
  • larsmans (1)
  • movermeyer (1)
  • desilinguist (1)
  • mulhod (1)
  • briverse17 (1)
Top Labels
Issue Labels
bug (2) enhancement (1) question (1)
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 2,635 last-month
  • Total docker downloads: 124
  • Total dependent packages: 0
  • Total dependent repositories: 10
  • Total versions: 6
  • Total maintainers: 3
pypi.org: match

Match tokenized words and phrases within the original, untokenized, often messy, text.

  • Versions: 6
  • Dependent Packages: 0
  • Dependent Repositories: 10
  • Downloads: 2,635 Last month
  • Docker Downloads: 124
Rankings
Docker downloads count: 2.6%
Dependent repos count: 4.6%
Downloads: 7.4%
Average: 8.7%
Dependent packages count: 10.0%
Stargazers count: 13.3%
Forks count: 14.2%
Maintainers (3)
desilinguist dmnapolitano tamarl08
Last synced: 12 months ago

Dependencies

requirements.txt pypi
  • nltk >=3.4
  • regex >=2018.7.11
.github/workflows/python-test.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
setup.py pypi