Autocorrect

Spelling corrector in python. Currently supports English, Polish, Turkish, Russian, Ukrainian, Czech, Portuguese, Greek, Italian, Vietnamese, French and Spanish, but you can easily add new languages.

Based on: https://github.com/phatpiglet/autocorrect and Peter Norvig's spelling corrector.

Installation

bash pip install autocorrect

Examples

Autocorrect full sentences:

```python

from autocorrect import Speller spell = Speller() spell("I'm not sleapy and tehre is no place I'm giong to.") "I'm not sleepy and there is no place I'm going to." ```

Use other languages:

```python

spell = Speller('pl') spell('ptaaki latatją kluczmm') 'ptaki latają kluczem' ```

Get multiple correction candidates for a single word:

```python

spell.get_candidates("tehre") [(5437024, 'there'), (5860, 'terre')] ``` The numbers are frequencies of a word, so the higher, the better.

Speed

python %timeit spell("I'm not sleapy and tehre is no place I'm giong to.") 373 µs ± 2.09 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) %timeit spell("There is no comin to consiousnes without pain.") 150 ms ± 2.02 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

As you see, for some words correction can take ~200ms. If speed is important for your use case (e.g. chatbot) you may want to use option 'fast': python spell = Speller(fast=True) %timeit spell("There is no comin to consiousnes without pain.") 344 µs ± 2.23 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) Now, the correction should always work in microseconds, but words with double typos (like 'consiousnes') won't be corrected.

OCR

When cleaning up OCR, replacements are the large majority of errors. If this is the case, you may want to use the option 'onlyreplacements': ```python spell = Speller(onlyreplacements=True) ```

Custom word sets

If you wish to use your own set of words for autocorrection, you can pass an nlp_data argument:

python spell = Speller(nlp_data=your_word_frequency_dict) Where your_word_frequency_dict is a dictionary which maps words to their average frequencies in your text. If you want to change the default word set only a bit, you can just edit spell.nlp_data parameter, after spell was initialized.

Adding new languages

A simpler but untested way - wordfreq

It should be possible to get word frequencies from the wordfreq package. You should be able to provide this word frequency data to autocorrector through nlp_data parameter. You will also need to generate appropriate alphabet (see constants.py).

A more complicated but tested way - wikipedia text

Note: I will no longer accept PRs to add individual languages using this method. A more sensible approach would be to try using the wordfreq method, and adding many languages at once in some general way. But I don't have time to implement this myself.

First, define special letters, by adding entries in word_regexes and alphabets dicts in autocorrect/constants.py.

Now, you need a bunch of text. Easiest way is to download wikipedia. For example for Russian you would go to: https://dumps.wikimedia.org/ruwiki/latest/ and download ruwiki-latest-pages-articles.xml.bz2

bzip2 -d ruiwiki-latest-pages-articles.xml.bz2

After that:

First, edit the autocorrect.constants dictionaries in order to accommodate regexes and dictionaries for your language.

Then:

```python

from autocorrect.wordcount import countwords count_words('ruwiki-latest-pages-articles.xml', 'ru') ```

tar -zcvf autocorrect/data/ru.tar.gz word_count.json

For the correction to work well, you need to cut out rarely used words. First, in testall.py, write test words for your language, and add them to optionallanguagetests the same way as it's done for other languages. It's good to have at least 30 words. Now run: ``` python testall.py find_threshold ru ``` and see which threshold value has the least badly corrected words. After that, manually delete all the words with less occurences than the threshold value you found, from the file in hi.tar.gz (it's already sorted so it should be easy).

To distribute this language support to others, you will need to upload your tar.gz file to IPFS (for example with Pinata, which will pin this file so it doesn't disappear), and then add it's path to ipfs_paths in constants.py. (tip: first put this file inside the folder, and upload the folder to IPFS, for the downloaded file to have the correct filename)

Good luck!