py-torchtext

Models, data loaders and abstractions for language processing, powered by PyTorch

https://github.com/pytorch/text

Science Score: 46.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
✓
Academic publication links
Links to: arxiv.org
✓
Committers with academic emails
8 of 156 committers (5.1%) from academic institutions
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (14.4%) to scientific vocabulary

Keywords

data-loader dataset deep-learning models nlp pytorch

Keywords from Contributors

tensor autograd audio speech transformer jax pretrained-models cryptography glm speech-recognition

Last synced: 6 months ago · JSON representation

Repository

Models, data loaders and abstractions for language processing, powered by PyTorch

Basic Info

Host: GitHub
Owner: pytorch
License: bsd-3-clause
Language: Python
Default Branch: main
Homepage: https://pytorch.org/text
Size: 47.3 MB

Statistics

Stars: 3,558
Watchers: 339
Forks: 815
Open Issues: 347
Releases: 31

Topics

data-loader dataset deep-learning models nlp pytorch

Created about 9 years ago · Last pushed 6 months ago

Metadata Files

Readme Contributing License Code of conduct

README.rst

.. image:: docs/source/_static/img/torchtext_logo.png

.. image:: https://circleci.com/gh/pytorch/text.svg?style=svg
    :target: https://circleci.com/gh/pytorch/text

.. image:: https://codecov.io/gh/pytorch/text/branch/main/graph/badge.svg
    :target: https://codecov.io/gh/pytorch/text

.. image:: https://img.shields.io/badge/dynamic/json.svg?label=docs&url=https%3A%2F%2Fpypi.org%2Fpypi%2Ftorchtext%2Fjson&query=%24.info.version&colorB=brightgreen&prefix=v
    :target: https://pytorch.org/text/

torchtext
+++++++++

**WARNING**: TorchText development is stopped and the `0.18` release (April 2024) will be the last stable release of the library.

This repository consists of:

* `torchtext.datasets `_: The raw text iterators for common NLP datasets
* `torchtext.data `_: Some basic NLP building blocks
* `torchtext.transforms `_: Basic text-processing transformations
* `torchtext.models `_: Pre-trained models
* `torchtext.vocab `_: Vocab and Vectors related classes and factory functions
* `examples `_: Example NLP workflows with PyTorch and torchtext library.


Installation
============

We recommend Anaconda as a Python package management system. Please refer to `pytorch.org `_ for the details of PyTorch installation. The following are the corresponding ``torchtext`` versions and supported Python versions.

.. csv-table:: Version Compatibility
   :header: "PyTorch version", "torchtext version", "Supported Python version"
   :widths: 10, 10, 10

   nightly build, main, ">=3.8, <=3.11"
   2.3.0, 0.18.0, ">=3.8, <=3.11"
   2.2.0, 0.17.0, ">=3.8, <=3.11"
   2.1.0, 0.16.0, ">=3.8, <=3.11"
   2.0.0, 0.15.0, ">=3.8, <=3.11"
   1.13.0, 0.14.0, ">=3.7, <=3.10"
   1.12.0, 0.13.0, ">=3.7, <=3.10"
   1.11.0, 0.12.0, ">=3.6, <=3.9"
   1.10.0, 0.11.0, ">=3.6, <=3.9"
   1.9.1, 0.10.1, ">=3.6, <=3.9"
   1.9, 0.10, ">=3.6, <=3.9"
   1.8.1, 0.9.1, ">=3.6, <=3.9"
   1.8, 0.9, ">=3.6, <=3.9"
   1.7.1, 0.8.1, ">=3.6, <=3.9"
   1.7, 0.8, ">=3.6, <=3.8"
   1.6, 0.7, ">=3.6, <=3.8"
   1.5, 0.6, ">=3.5, <=3.8"
   1.4, 0.5, "2.7, >=3.5, <=3.8"
   0.4 and below, 0.2.3, "2.7, >=3.5, <=3.8"

Using conda::

    conda install -c pytorch torchtext

Using pip::

    pip install torchtext

Optional requirements
---------------------

If you want to use English tokenizer from `SpaCy `_, you need to install SpaCy and download its English model::

    pip install spacy
    python -m spacy download en_core_web_sm

Alternatively, you might want to use the `Moses `_ tokenizer port in `SacreMoses `_ (split from `NLTK `_). You have to install SacreMoses::

    pip install sacremoses

For torchtext 0.5 and below, ``sentencepiece``::

    conda install -c powerai sentencepiece

Building from source
--------------------

To build torchtext from source, you need ``git``, ``CMake`` and C++11 compiler such as ``g++``.::

    git clone https://github.com/pytorch/text torchtext
    cd torchtext
    git submodule update --init --recursive

    # Linux
    python setup.py clean install

    # OSX
    CC=clang CXX=clang++ python setup.py clean install

    # or ``python setup.py develop`` if you are making modifications.

**Note**

When building from source, make sure that you have the same C++ compiler as the one used to build PyTorch. A simple way is to build PyTorch from source and use the same environment to build torchtext.
If you are using the nightly build of PyTorch, checkout the environment it was built with `conda (here) `_ and `pip (here) `_.

Additionally, datasets in torchtext are implemented using the torchdata library. Please take a look at the
`installation instructions `_ to download the latest nightlies or install from source.

Documentation
=============

Find the documentation `here `_.

Datasets
========

The datasets module currently contains:

* Language modeling: WikiText2, WikiText103, PennTreebank, EnWik9
* Machine translation: IWSLT2016, IWSLT2017, Multi30k
* Sequence tagging (e.g. POS/NER): UDPOS, CoNLL2000Chunking
* Question answering: SQuAD1, SQuAD2
* Text classification: SST2, AG_NEWS, SogouNews, DBpedia, YelpReviewPolarity, YelpReviewFull, YahooAnswers, AmazonReviewPolarity, AmazonReviewFull, IMDB
* Model pre-training: CC-100

Models
======

The library currently consist of following pre-trained models:

* RoBERTa: `Base and Large Architecture `_
* `DistilRoBERTa `_
* XLM-RoBERTa: `Base and Large Architure `_
* T5: `Small, Base, Large, 3B, and 11B Architecture `_
* Flan-T5: `Base, Large, XL, and XXL Architecture `_

Tokenizers
==========

The transforms module currently support following scriptable tokenizers:

* `SentencePiece `_
* `GPT-2 BPE `_
* `CLIP `_
* `RE2 `_
* `BERT `_

Tutorials
=========

To get started with torchtext, users may refer to the following tutorial available on PyTorch website.

* `SST-2 binary text classification using XLM-R pre-trained model `_
* `Text classification with AG_NEWS dataset `_
* `Translation trained with Multi30k dataset using transformers and torchtext `_
* `Language modeling using transforms and torchtext `_


Disclaimer on Datasets
======================

This is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use the dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license.

If you're a dataset owner and wish to update any part of it (description, citation, etc.), or do not want your dataset to be included in this library, please get in touch through a GitHub issue. Thanks for your contribution to the ML community!

Owner

Name: pytorch
Login: pytorch
Kind: organization
Location: where the eigens are valued

Website: https://pytorch.org
Repositories: 83
Profile: https://github.com/pytorch

GitHub Events

Total

Issues event: 19
Watch event: 70
Issue comment event: 33
Push event: 268
Pull request review event: 1
Pull request event: 6
Fork event: 17

Last Year

Issues event: 19
Watch event: 70
Issue comment event: 33
Push event: 268
Pull request review event: 1
Pull request event: 6
Fork event: 17

Committers

Last synced: 9 months ago

All Time

Total Commits: 1,268
Total Committers: 156
Avg Commits per committer: 8.128
Development Distribution Score (DDS): 0.864

Past Year

Commits: 3
Committers: 2
Avg Commits per committer: 1.5
Development Distribution Score (DDS): 0.333

Top Committers

Name	Email	Commits
Nayef Ahmed	2****1	172
parmeet	p**a@f**m	164
Guanheng George Zhang	6****6	128
Nelson Liu	n**u@c**u	113
jekbradbury	j**y@g**m	79
cpuhrsch	c**h@f**m	43
moto	8****k	42
Martin Tutek	m**k@g**m	37
Elijah Rippeth	e**h@g**m	36
Andrey Talman	a**n@f**m	32
Bryan Marcus McCann	B**s@g**m	31
Omkar Salpekar	o**r@f**m	28
Joe Cummings	j**s@f**m	27
Abhinav Arora	a**a@f**m	19
VirgileHlav	v**m@f**m	19
pmabbo13	8****3	17
Eli Uriegas	1****e	15
Nicolas Hug	c**t@n**m	12
Kento Nozawa	k**w@k**p	11
Roman Shraga	r**a@g**m	9
Nikita Shulga	n**a@f**m	8
keitakurita	k**a@g**m	8
Kevin Tse	N****T	8
peterjc123	p**6@g**m	7
Guanheng Zhang	z**g@d**r	7
Matti Picus	m**s@g**m	6
Sumit Kumar	s**r@g**m	5
ProGamerGov	P****v	5
Mustafa Bal	5****l	4
Jihun Choi	j**i@e**r	4
and 126 more...

Committer Domains (Top 20 + Academic)

fb.com: 15 devfair0197.h2.fair: 2 meta.com: 2 cs.washington.edu: 1 nicolas-hug.com: 1 klis.tsukuba.ac.jp: 1 europa.snu.ac.kr: 1 traveloka.com: 1 nyu.edu: 1 kkurniawan.com: 1 sivareddy.in: 1 fer.hr: 1 herhjemme.dk: 1 nextbsd.org: 1 rit.edu: 1 uni-wuerzburg.de: 1 wisc.edu: 1 digitalreasoning.com: 1 smerity.com: 1 qq.com: 1 uic.edu: 1 mail.mcgill.ca: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 113
Total pull requests: 160
Average time to close issues: 7 months
Average time to close pull requests: 6 months
Total issue authors: 98
Total pull request authors: 46
Average comments per issue: 2.35
Average comments per pull request: 1.49
Merged pull requests: 90
Bot issues: 0
Bot pull requests: 1

Past Year

Issues: 17
Pull requests: 7
Average time to close issues: about 1 month
Average time to close pull requests: 1 minute
Issue authors: 17
Pull request authors: 5
Average comments per issue: 0.41
Average comments per pull request: 1.0
Merged pull requests: 1
Bot issues: 0
Bot pull requests: 1

View more stats

Top Authors

Issue Authors

joecummings (7)
saeeddhqan (3)
AlexanderTrg (2)
ReamzZ (2)
h-vetinari (2)
mullerhai (2)
jiapei100 (2)
premiitr (2)
gchaperon (2)
lkxr2k (1)
cdeepali (1)
TunifyBasic (1)
AnthoJack (1)
cqray1990 (1)
adzcai (1)

Pull Request Authors

atalman (25)
NicolasHug (20)
Nayef211 (14)
joecummings (11)
PaliC (9)
osalpekar (8)
huydhn (5)
DanilBaibak (4)
wangshuai09 (4)
cpuhrsch (4)
kit1980 (3)
parmeet (3)
SM-Jang (3)
mthrok (3)
johnnynunez (2)

Top Labels

Issue Labels

feature request (4) enhancement (3) bug (3) legacy (2) datasets (2) Good for the first PR contribution (1) help wanted (1) testing (1) BE hackathon (1) need discussions (1) good first issue (1) cla signed (1) obsolete (1)

Pull Request Labels

cla signed (142) fb-exported (3) obsolete (3) dependencies (2) github_actions (2) enhancement (1) docs (1) BE hackathon (1) with-ssh (1) module: rocm (1) ciflow/default (1)

Packages

Total packages: 3
Total downloads:
- pypi 647,692 last-month
Total docker downloads: 35,666,527

Total dependent packages: 93
(may contain duplicates)
Total dependent repositories: 2,976
(may contain duplicates)
Total versions: 131
Total maintainers: 5

pypi.org: torchtext

Text utilities, models, transforms, and datasets for PyTorch.

Homepage: https://github.com/pytorch/text
Documentation: https://torchtext.readthedocs.io/
License: BSD
Latest release: 0.18.0
published almost 2 years ago

Versions: 33
Dependent Packages: 92
Dependent Repositories: 2,976
Downloads: 647,692 Last month
Docker Downloads: 35,666,527

Rankings

Dependent repos count: 0.2%

Dependent packages count: 0.2%

Downloads: 0.5%

Docker downloads count: 0.6%

Average: 0.7%

Stargazers count: 1.3%

Forks count: 1.5%

Maintainers (4)

soumith seemethere malfet atalman

Last synced: 6 months ago

proxy.golang.org: github.com/pytorch/text

Documentation: https://pkg.go.dev/github.com/pytorch/text#section-documentation
License: bsd-3-clause
Latest release: v0.18.0
published almost 2 years ago

Versions: 97
Dependent Packages: 0
Dependent Repositories: 0

Rankings

Dependent packages count: 7.0%

Average: 8.2%

Dependent repos count: 9.3%

Last synced: 6 months ago

spack.io: py-torchtext

Text utilities, models, transforms, and datasets for PyTorch.

Homepage: https://github.com/pytorch/text
License: []
Latest release: 0.5.0
published almost 4 years ago

Versions: 1
Dependent Packages: 1
Dependent Repositories: 0

Rankings

Dependent repos count: 0.0%

Forks count: 3.1%

Stargazers count: 4.0%

Average: 8.8%

Dependent packages count: 28.1%

Maintainers (1)

adamjstewart

Last synced: 6 months ago

Dependencies

.github/workflows/bandit.yml actions

actions/checkout v2 composite

.github/workflows/build-conda-linux.yml actions

.github/workflows/build-conda-m1.yml actions

.github/workflows/build-conda-macos.yml actions

.github/workflows/build-conda-windows.yml actions

.github/workflows/build-docs.yml actions

actions/checkout v3 composite
actions/download-artifact v3 composite

.github/workflows/build-wheels-linux.yml actions

.github/workflows/build-wheels-m1.yml actions

.github/workflows/build-wheels-macos.yml actions

.github/workflows/build-wheels-windows.yml actions

.github/workflows/codeql.yml actions

actions/checkout v2 composite
github/codeql-action/analyze v1 composite
github/codeql-action/init v1 composite

.github/workflows/integration-test.yml actions

.github/workflows/lint.yml actions

.github/workflows/test-linux-cpu.yml actions

.github/workflows/test-linux-gpu.yml actions

.github/workflows/test-macos-cpu.yml actions

.github/workflows/test-windows-cpu.yml actions

.github/workflows/validate-binaries.yml actions

.github/workflows/validate-nightly-binaries.yml actions

docs/requirements.txt pypi

Jinja2 <3.1.0
matplotlib *
regex *
sphinx ==5.1.1
sphinx_gallery ==0.11.1

pyproject.toml pypi

requirements.txt pypi

Sphinx *
codecov *
defusedxml *
expecttest *
nltk *
parameterized *
pytest *
pytest-cov *
pytest-pythonpath *
requests *
sacremoses *
spacy *
torchdata >0.5
tqdm *

setup.py pypi

tqdm *

.circleci/unittest/linux/scripts/environment.yml conda

codecov
pip

.circleci/unittest/windows/scripts/environment.yml conda

codecov
pip
setuptools 58.0.4
spacy