subs2vec

Tools for training and evaluating word embeddings based on subtitles. Published as "subs2vec: Word embeddings from subtitles in 55 languages" in Behavior Research Methods.

https://github.com/jvparidon/subs2vec

Last synced: 11 months ago · JSON representation

Repository

Tools for training and evaluating word embeddings based on subtitles. Published as "subs2vec: Word embeddings from subtitles in 55 languages" in Behavior Research Methods.

Basic Info

Host: GitHub
Owner: jvparidon
License: mit
Language: Python
Default Branch: master
Homepage: https://doi.org/10.3758/s13428-020-01406-3
Size: 226 MB

Statistics

Stars: 30
Watchers: 2
Forks: 5
Open Issues: 1
Releases: 0

Created over 8 years ago · Last pushed almost 6 years ago

Metadata Files

Readme License

subs2vec

Van Paridon & Thompson (2019) introduces pretrained embeddings and precomputed word/bigram/trigram frequencies in 55 languages. The files can be downloaded from the links in this table. Word vectors trained on subtitles are available, as well as vectors trained on Wikipedia, and a combination of subtitles and Wikipedia (for best predictive performance).

This repository contains the subs2vec module, a number of Python 3.7 scripts and command line tools to evaluate a set of word vectors on semantic similarity, semantic and syntactic analogy, and lexical norm prediction tasks. In addition, the subs2vec.py script will take an OpenSubtitles archive or Wikipedia and go through all the steps to train a fastText model and produce word vectors as used in the paper associated with this repository.

Psycholinguists may be especially interested norms script, which evaluates the lexical norm prediction performance of a set of word vectors, but can also be used to predict lexical norms for un-normed words. For a more detailed explanation see the how to use -> extending lexical norms section.

The scripts in this repository require Python 3.7 and some additional libraries that are easily installed through pip. (If you want to use the subs2vec.py script to train your own word embeddings, you will also need compiled fastText and word2vec binaries.)

If you use any of the subs2vec code and/or pretrained models, please cite the preprint (Van Paridon & Thompson, 2019).

How to use

subs2vec is available through pip, installing is as easy as running:
python3 -m pip install subs2vec
Any missing dependencies should be installed automatically.

Each submodules of subs2vec can then be run as a command line tool using the -m flag:
python3 -m subs2vec.submodule_name

Evaluating word embeddings

To evaluate word embeddings on analogies, semantic similarity, or lexical norm prediction as in Van Paridon & Thompson (2019), use:
python3 -m subs2vec.analogies fr french_word_vectors.vec
python3 -m subs2vec.similarities fr french_word_vectors.vec
python3 -m subs2vec.norms fr french_word_vectors.vec
subs2vec uses the two-letter ISO language codes, so French in the example is fr, English would be en, German would be de, etc.

All datasets used for evaluation, including the lexical norms, are stored in subs2vec/evaluation/datasets/.
Results from Van Paridon & Thompson (2019) are in subs2vec/evaluation/article_results/.

Extending lexical norms

To extend lexical norms (either norms you have collected yourself, or norms provided in this repository) use:
python3 -m subs2vec.norms fr french_word_vectors.vec --extend_norms=french_norms_file.txt

The norms file should be a tab-separated text file, with the first line containing column names and the column containing the words should be called word. Unobserved cells should be left empty. If you are unsure how to generate this file, you can create your list in Excel and then use Save as... tab-delimited text.
For an overview of norms that come included in the repo (and their authors), see this list. For the norms datasets themselves, look inside this directory.

Extracting word frequencies

The subtitle corpus used to train subs2vec was also used to compile the word frequencies in SUBTLEX. That same corpus can of course be used to compile bigram and trigram frequencies as well.
To extract word, bigram, or trigram frequencies from a text file yourself, fr.txt for instance, use:
python3 -m subs2vec.frequencies fr.txt

In general, however, we recommend downloading the precompiled frequencies files from [language archive] and looking frequencies up in those.
When looking up frequencies for specific words, bigrams, or trigrams, you may find that you cannot open the frequencies files (they can be very large). To retrieve items of interest use:
python3 -m subs2vec.lookup frequencies_file.tsv list_of_items.txt
Your list of items should be a simple text file, with each item you want to look up on its own line.
This lookup scripts works for looking up frequencies, but it finds lines in any plain text file, so it works for looking up word vectors in .vec files as well.

Removing duplicate lines

subs2vec comes with a module that removes duplicate lines from text files. We used it to remove duplicate lines from training corpora, but it works for any text file.
To remove duplicates from fr.txt for example, use:
python3 -m subs2vec.deduplicate fr.txt

Training models

If you want to reproduce models as used in Van Paridon & Thompson (2019), you can use the train_model module.
For instance, the steps to create a subtitle corpus are: 1. Download a corpus:
python3 -m subs2vec.download fr subs
2. Clean the corpus:
python3 -m subs2vec.clean_subs fr --strip --join
3. Deduplicate the lines in the corpus:
python3 -m subs2vec.deduplicate fr.txt
4. Train a fastText model on the subtitle corpus:
python3 -m subs2vec.train_model fr subs dedup.fr.txt
This last step requires the binaries for fastText and word2phrase (part of word2vec) to be downloaded, built, and discoverable on your system (i.e., on your PATH).

For more detailed training options:
python3 -m subs2vec.train_model --help

API

For more detailed documentation of the package modules and API, see subs2vec.readthedocs.io

Downloading datasets

This table contains links to the top 1 million word vectors in each language, as well all vectors, model binaries, and the word, bigram, and trigram frequencies in the subtitle and Wikipedia corpora. If you use these pretrained vectors/models, please cite the preprint (Van Paridon & Thompson, 2019).

Owner

Name: JP van Paridon
Login: jvparidon
Kind: user
Location: Madison, Wisconsin
Company: University of Wisconsin-Madison

Website: jvparidon.io
Twitter: jvparidon
Repositories: 2
Profile: https://github.com/jvparidon

Data scientist; postdoc at the University of Wisconsin-Madison. Using experimental, statistical, and computational methods to study language.

GitHub Events

Total

Watch event: 1

Last Year

Watch event: 1

Committers

Last synced: over 3 years ago

All Time

Total Commits: 219
Total Committers: 4
Avg Commits per committer: 54.75
Development Distribution Score (DDS): 0.47

Top Committers

Name	Email	Commits
JvParidon	j**n@g**m	116
Jeroen van Paridon	J**n@m**l	99
Jeroen van Paridon	j**r@m**l	3
bill	b**l@m**l	1

Committer Domains (Top 20 + Academic)

mpi.nl: 2

Issues and Pull Requests

Last synced: over 2 years ago

All Time

Total issues: 4
Total pull requests: 0
Average time to close issues: 3 days
Average time to close pull requests: N/A
Total issue authors: 4
Total pull request authors: 0
Average comments per issue: 1.75
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 0
Pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Issue authors: 0
Pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

nilsymbol (1)
weiguowilliam (1)
fotisj (1)
mdingemanse (1)

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

Total packages: 1
Total downloads:
- pypi 16 last-month

Total dependent packages: 0
Total dependent repositories: 1
Total versions: 3
Total maintainers: 1

pypi.org: subs2vec

subs2vec module

Homepage: https://github.com/jvparidon/subs2vec/
Documentation: https://subs2vec.readthedocs.io/
License: MIT
Latest release: 0.9.6
published over 6 years ago

Versions: 3
Dependent Packages: 0
Dependent Repositories: 1
Downloads: 16 Last month

Rankings

Dependent packages count: 10.0%

Stargazers count: 11.4%

Forks count: 14.2%

Average: 20.5%

Dependent repos count: 21.7%

Downloads: 44.9%

Maintainers (1)

jvparidon

Last synced: 11 months ago

subs2vec

Science Score: 23.0%

Repository

Basic Info

Statistics

Metadata Files

README.md

subs2vec

How to use

Evaluating word embeddings

Extending lexical norms

Extracting word frequencies

Removing duplicate lines

Training models

API

Downloading datasets

Owner

GitHub Events

Total

Last Year

Committers

All Time

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: subs2vec

Rankings

Maintainers (1)

Dependencies