Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of gensim

gensim -

Changes

4.3.2, 2023-08-23

:red_circle: Bug fixes

  • Fix incorrect conversion of cosine distance to cosine similarity (monash849, #3441)

:books: Tutorial and doc improvements

:+1: Improvements

🔮 Testing, CI, housekeeping

- Python
Published by mpenkov almost 3 years ago

gensim - 4.3.0

What's Changed

  • Allow overriding the Cython version requirement by @pabs3 in https://github.com/RaRe-Technologies/gensim/pull/3323
  • Update Python module MANIFEST by @pabs3 in https://github.com/RaRe-Technologies/gensim/pull/3343
  • Clean up references to Morfessor, tox and gensim.models.wrappers by @pabs3 in https://github.com/RaRe-Technologies/gensim/pull/3345
  • Disable the Gensim 3=>4 warning in docs by @piskvorky in https://github.com/RaRe-Technologies/gensim/pull/3346
  • pin sphinx versions, add explicit gallery_top label by @mpenkov in https://github.com/RaRe-Technologies/gensim/pull/3383
  • Declare variables prior to for loop in fastss.pyx for ANSI C compatibility by @hstk30 in https://github.com/RaRe-Technologies/gensim/pull/3378
  • Fix typo in word2vec and KeyedVectors docstrings by @dymil in https://github.com/RaRe-Technologies/gensim/pull/3365
  • Replace np.multiply with np.square and copyedit in translation_matrix.py by @dymil in https://github.com/RaRe-Technologies/gensim/pull/3374
  • Copyedit and fix outdated statements in translation matrix tutorial by @dymil in https://github.com/RaRe-Technologies/gensim/pull/3375
  • Implement Okapi BM25 variants in Gensim by @Witiko in https://github.com/RaRe-Technologies/gensim/pull/3304
  • Giving missing credit in EnsembleLDA to Alex in docs by @sezanzeb in https://github.com/RaRe-Technologies/gensim/pull/3393
  • PERF: pyemd to POT for EMD computation in wmdistance by @TLouf in https://github.com/RaRe-Technologies/gensim/pull/3327
  • Fixed bug in loss computation for Word2Vec with hierarchical softmax by @TalIfargan in https://github.com/RaRe-Technologies/gensim/pull/3397
  • fix deprecation warning from pytest by @martino-vic in https://github.com/RaRe-Technologies/gensim/pull/3354
  • Switch to Cython language level 3 by @pabs3 in https://github.com/RaRe-Technologies/gensim/pull/3344
  • Implement numpy hack in setup.py to enable install under Poetry by @jaymegordo in https://github.com/RaRe-Technologies/gensim/pull/3363
  • Fixed the broken link in readme.md by @aswin2108 in https://github.com/RaRe-Technologies/gensim/pull/3409
  • Path Coherence Model to correctly handle empty documents by @PrimozGodec in https://github.com/RaRe-Technologies/gensim/pull/3406
  • Add support for Python 3.11 and drop support for Python 3.7 by @acul3 in https://github.com/RaRe-Technologies/gensim/pull/3402
  • clarify runtime expectations by @gojomo in https://github.com/RaRe-Technologies/gensim/pull/3381
  • Fix bug that prevents loading old models by @funasshi in https://github.com/RaRe-Technologies/gensim/pull/3359
  • refactor wheel building and testing workflow by @mpenkov in https://github.com/RaRe-Technologies/gensim/pull/3410
  • Fixed FastTextKeyedVectors handling in add_vector by @globba in https://github.com/RaRe-Technologies/gensim/pull/3389
  • Flsamodel by @ERijck in https://github.com/RaRe-Technologies/gensim/pull/3398
  • Fix backwards compatibility bug in Word2Vec by @mpenkov in https://github.com/RaRe-Technologies/gensim/pull/3415
  • fix numpy hack in setup.py by @mpenkov in https://github.com/RaRe-Technologies/gensim/pull/3416
  • updated changelog for next release by @mpenkov in https://github.com/RaRe-Technologies/gensim/pull/3412

New Contributors

  • @hstk30 made their first contribution in https://github.com/RaRe-Technologies/gensim/pull/3378
  • @TLouf made their first contribution in https://github.com/RaRe-Technologies/gensim/pull/3327
  • @TalIfargan made their first contribution in https://github.com/RaRe-Technologies/gensim/pull/3397
  • @martino-vic made their first contribution in https://github.com/RaRe-Technologies/gensim/pull/3354
  • @jaymegordo made their first contribution in https://github.com/RaRe-Technologies/gensim/pull/3363
  • @aswin2108 made their first contribution in https://github.com/RaRe-Technologies/gensim/pull/3409
  • @acul3 made their first contribution in https://github.com/RaRe-Technologies/gensim/pull/3402
  • @funasshi made their first contribution in https://github.com/RaRe-Technologies/gensim/pull/3359
  • @globba made their first contribution in https://github.com/RaRe-Technologies/gensim/pull/3389
  • @ERijck made their first contribution in https://github.com/RaRe-Technologies/gensim/pull/3398

Full Changelog: https://github.com/RaRe-Technologies/gensim/compare/4.2.0...4.3.0

- Python
Published by mpenkov over 3 years ago

gensim - 4.2.0

A number of incremental improvements, optimizations and bugfixes: CHANGELOG

- Python
Published by piskvorky about 4 years ago

gensim -

4.1.2, 2021-09-17

This is a bugfix release that addresses left over compatibility issues with older versions of numpy and MacOS.

4.1.1, 2021-09-14

This is a bugfix release that addresses compatibility issues with older versions of numpy.

4.1.0, 2021-08-15

Gensim 4.1 brings two major new functionalities:

There are several minor changes that are not backwards compatible with previous versions of Gensim. The affected functionality is relatively less used, so it is unlikely to affect most users, so we have opted to not require a major version bump. Nevertheless, we describe them below.

Improved parameter edge-case handling in KeyedVectors mostsimilar and mostsimilar_cosmul methods

We now handle both positive and negative keyword parameters consistently. They may now be either:

  1. A string, in which case the value is reinterpreted as a list of one element (the string value)
  2. A vector, in which case the value is reinterpreted as a list of one element (the vector)
  3. A list of strings
  4. A list of vectors

So you can now simply do:

python model.most_similar(positive='war', negative='peace')

instead of the slightly more involved

python model.most_similar(positive=['war'], negative=['peace'])

Both invocations remain correct, so you can use whichever is most convenient. If you were somehow expecting gensim to interpret the strings as a list of characters, e.g.

python model.most_similar(positive=['w', 'a', 'r'], negative=['p', 'e', 'a', 'c', 'e'])

then you will need to specify the lists explicitly in gensim 4.1.

Deprecated obsolete step parameter from doc2vec

With the newer version, do this:

python model.infer_vector(..., epochs=123)

instead of this:

python model.infer_vector(..., steps=123)

Plus a large number of smaller improvements and fixes, as usual.

⚠️ If migrating from old Gensim 3.x, read the Migration guide first.

:+1: New features

  • #3169: Implement shrink_windows argument for Word2Vec, by @M-Demay
  • #3163: Optimize word mover distance (WMD) computation, by @flowlight0
  • #3157: New KeyedVectors.vectorsforall method for vectorizing all words in a dictionary, by @Witiko
  • #3153: Vectorize word2vec.predictoutputword for speed, by @M-Demay
  • #3146: Use FastSS for fast kNN over Levenshtein distance, by @Witiko
  • #3128: Materialize and copy the corpus passed to SoftCosineSimilarity, by @Witiko
  • #3115: Make LSI dispatcher CLI param for number of jobs optional, by @robguinness
  • #3091: LsiModel: Only log top words that actually exist in the dictionary, by @kmurphy4
  • #2980: Added EnsembleLda for stable LDA topics, by @sezanzeb
  • #2978: Optimize performance of Author-Topic model, by @horpto
  • #3000: Tidy up KeyedVectors.most_similar() API, by @simonwiles

:books: Tutorials and docs

:red_circle: Bug fixes

  • #3178: Fix Unicode string incompatibility in gensim.similarities.fastss.editdist, by @Witiko
  • #3174: Fix loading Phraser models stored in Gensim 3.x into Gensim 4.0, by @emgucv
  • #3136: Fix indexing error in word2vec_inner.pyx, by @bluekura
  • #3131: Add missing import to NMF docs and models/init.py, by @properGrammar
  • #3116: Fix bug where saved Phrases model did not load its connector_words, by @aloknayak29
  • #2830: Fixed KeyError in coherence model, by @pietrotrope

:warning: Removed functionality & deprecations

  • #3176: Eliminate obsolete step parameter from doc2vec infervector and similarityunseen_docs, by @rock420
  • #2965: Remove strippunctuation2 alias of strippunctuation, by @sciatro
  • #3180: Move preprocessing functions from gensim.corpora.textcorpus and gensim.corpora.lowcorpus to gensim.parsing.preprocessing, by @rock420

🔮 Testing, CI, housekeeping

  • #3156: Update Numpy minimum version to 1.17.0, by @PrimozGodec
  • #3143: replace _mul function with explicit casts, by @mpenkov
  • #2952: Allow newer versions of the Morfessor module for the tests, by @pabs3
  • #2965: Remove strippunctuation2 alias of strippunctuation, by @sciatro

- Python
Published by mpenkov over 4 years ago

gensim -

4.1.1, 2021-09-14

This is a bugfix release that addresses compatibility issues with older versions of numpy.

4.1.0, 2021-08-15

Gensim 4.1 brings two major new functionalities:

There are several minor changes that are not backwards compatible with previous versions of Gensim. The affected functionality is relatively less used, so it is unlikely to affect most users, so we have opted to not require a major version bump. Nevertheless, we describe them below.

Improved parameter edge-case handling in KeyedVectors mostsimilar and mostsimilar_cosmul methods

We now handle both positive and negative keyword parameters consistently. They may now be either:

  1. A string, in which case the value is reinterpreted as a list of one element (the string value)
  2. A vector, in which case the value is reinterpreted as a list of one element (the vector)
  3. A list of strings
  4. A list of vectors

So you can now simply do:

python model.most_similar(positive='war', negative='peace')

instead of the slightly more involved

python model.most_similar(positive=['war'], negative=['peace'])

Both invocations remain correct, so you can use whichever is most convenient. If you were somehow expecting gensim to interpret the strings as a list of characters, e.g.

python model.most_similar(positive=['w', 'a', 'r'], negative=['p', 'e', 'a', 'c', 'e'])

then you will need to specify the lists explicitly in gensim 4.1.

Deprecated obsolete step parameter from doc2vec

With the newer version, do this:

python model.infer_vector(..., epochs=123)

instead of this:

python model.infer_vector(..., steps=123)

Plus a large number of smaller improvements and fixes, as usual.

⚠️ If migrating from old Gensim 3.x, read the Migration guide first.

:+1: New features

  • #3169: Implement shrink_windows argument for Word2Vec, by @M-Demay
  • #3163: Optimize word mover distance (WMD) computation, by @flowlight0
  • #3157: New KeyedVectors.vectorsforall method for vectorizing all words in a dictionary, by @Witiko
  • #3153: Vectorize word2vec.predictoutputword for speed, by @M-Demay
  • #3146: Use FastSS for fast kNN over Levenshtein distance, by @Witiko
  • #3128: Materialize and copy the corpus passed to SoftCosineSimilarity, by @Witiko
  • #3115: Make LSI dispatcher CLI param for number of jobs optional, by @robguinness
  • #3091: LsiModel: Only log top words that actually exist in the dictionary, by @kmurphy4
  • #2980: Added EnsembleLda for stable LDA topics, by @sezanzeb
  • #2978: Optimize performance of Author-Topic model, by @horpto
  • #3000: Tidy up KeyedVectors.most_similar() API, by @simonwiles

:books: Tutorials and docs

:red_circle: Bug fixes

  • #3178: Fix Unicode string incompatibility in gensim.similarities.fastss.editdist, by @Witiko
  • #3174: Fix loading Phraser models stored in Gensim 3.x into Gensim 4.0, by @emgucv
  • #3136: Fix indexing error in word2vec_inner.pyx, by @bluekura
  • #3131: Add missing import to NMF docs and models/init.py, by @properGrammar
  • #3116: Fix bug where saved Phrases model did not load its connector_words, by @aloknayak29
  • #2830: Fixed KeyError in coherence model, by @pietrotrope

:warning: Removed functionality & deprecations

  • #3176: Eliminate obsolete step parameter from doc2vec infervector and similarityunseen_docs, by @rock420
  • #2965: Remove strippunctuation2 alias of strippunctuation, by @sciatro
  • #3180: Move preprocessing functions from gensim.corpora.textcorpus and gensim.corpora.lowcorpus to gensim.parsing.preprocessing, by @rock420

🔮 Testing, CI, housekeeping

  • #3156: Update Numpy minimum version to 1.17.0, by @PrimozGodec
  • #3143: replace _mul function with explicit casts, by @mpenkov
  • #2952: Allow newer versions of the Morfessor module for the tests, by @pabs3
  • #2965: Remove strippunctuation2 alias of strippunctuation, by @sciatro

- Python
Published by mpenkov over 4 years ago

gensim - 4.1.0

4.1.0, 2021-08-15

Gensim 4.1 brings two major new functionalities:

There are several minor changes that are not backwards compatible with previous versions of Gensim. The affected functionality is relatively less used, so it is unlikely to affect most users, so we have opted to not require a major version bump. Nevertheless, we describe them below.

Improved parameter edge-case handling in KeyedVectors mostsimilar and mostsimilar_cosmul methods

We now handle both positive and negative keyword parameters consistently. They may now be either:

  1. A string, in which case the value is reinterpreted as a list of one element (the string value)
  2. A vector, in which case the value is reinterpreted as a list of one element (the vector)
  3. A list of strings
  4. A list of vectors

So you can now simply do:

python model.most_similar(positive='war', negative='peace')

instead of the slightly more involved

python model.most_similar(positive=['war'], negative=['peace'])

Both invocations remain correct, so you can use whichever is most convenient. If you were somehow expecting gensim to interpret the strings as a list of characters, e.g.

python model.most_similar(positive=['w', 'a', 'r'], negative=['p', 'e', 'a', 'c', 'e'])

then you will need to specify the lists explicitly in gensim 4.1.

Deprecated obsolete step parameter from doc2vec

With the newer version, do this:

python model.infer_vector(..., epochs=123)

instead of this:

python model.infer_vector(..., steps=123)

Plus a large number of smaller improvements and fixes, as usual.

⚠️ If migrating from old Gensim 3.x, read the Migration guide first.

:+1: New features

  • #3169: Implement shrink_windows argument for Word2Vec, by @M-Demay
  • #3163: Optimize word mover distance (WMD) computation, by @flowlight0
  • #3157: New KeyedVectors.vectorsforall method for vectorizing all words in a dictionary, by @Witiko
  • #3153: Vectorize word2vec.predictoutputword for speed, by @M-Demay
  • #3146: Use FastSS for fast kNN over Levenshtein distance, by @Witiko
  • #3128: Materialize and copy the corpus passed to SoftCosineSimilarity, by @Witiko
  • #3115: Make LSI dispatcher CLI param for number of jobs optional, by @robguinness
  • #3091: LsiModel: Only log top words that actually exist in the dictionary, by @kmurphy4
  • #2980: Added EnsembleLda for stable LDA topics, by @sezanzeb
  • #2978: Optimize performance of Author-Topic model, by @horpto
  • #3000: Tidy up KeyedVectors.most_similar() API, by @simonwiles

:books: Tutorials and docs

:red_circle: Bug fixes

  • #3178: Fix Unicode string incompatibility in gensim.similarities.fastss.editdist, by @Witiko
  • #3174: Fix loading Phraser models stored in Gensim 3.x into Gensim 4.0, by @emgucv
  • #3136: Fix indexing error in word2vec_inner.pyx, by @bluekura
  • #3131: Add missing import to NMF docs and models/init.py, by @properGrammar
  • #3116: Fix bug where saved Phrases model did not load its connector_words, by @aloknayak29
  • #2830: Fixed KeyError in coherence model, by @pietrotrope

:warning: Removed functionality & deprecations

  • #3176: Eliminate obsolete step parameter from doc2vec infervector and similarityunseen_docs, by @rock420
  • #2965: Remove strippunctuation2 alias of strippunctuation, by @sciatro
  • #3180: Move preprocessing functions from gensim.corpora.textcorpus and gensim.corpora.lowcorpus to gensim.parsing.preprocessing, by @rock420

🔮 Testing, CI, housekeeping

  • #3156: Update Numpy minimum version to 1.17.0, by @PrimozGodec
  • #3143: replace _mul function with explicit casts, by @mpenkov
  • #2952: Allow newer versions of the Morfessor module for the tests, by @pabs3
  • #2965: Remove strippunctuation2 alias of strippunctuation, by @sciatro

4.0.1, 2021-04-01

Bugfix release to address issues with Wheels on Windows:

  • https://github.com/RaRe-Technologies/gensim/issues/3095
  • https://github.com/RaRe-Technologies/gensim/issues/3097

4.0.0, 2021-03-24

⚠️ Gensim 4.0 contains breaking API changes! See the Migration guide to update your existing Gensim 3.x code and models.

Gensim 4.0 is a major release with lots of performance & robustness improvements, and a new website.

Main highlights

a. Efficiency

| model | 3.8.3: wall time / peak RAM / throughput | 4.0.0: wall time / peak RAM / throughput |
|----------|------------|--------|
| fastText | 2.9h / 4.11 GB / 822k words/s | 2.3h / **1.26 GB** / 914k words/s |
| word2vec | 1.7h / 0.36 GB / 1685k words/s | **1.2h** / 0.33 GB / 1762k words/s |

In other words, fastText now needs 3x less RAM (and is faster); word2vec has 2x faster init (and needs less RAM, and is faster); detecting collocation phrases is 2x faster. ([4.0 benchmarks](https://github.com/RaRe-Technologies/gensim/issues/2887#issuecomment-711097334))

b. Robustness. We fixed a bunch of long-standing bugs by refactoring the internal code structure (see 🔴 Bug fixes below)

c. Simplified OOP model for easier model exports and integration with TensorFlow, PyTorch &co.

These improvements come to you transparently aka "for free", but see Migration guide for some changes that break the old Gensim 3.x API. Update your code accordingly.

  • Dropped a bunch of externally contributed modules and wrappers: summarization, pivoted TFIDF, Mallet…

    • Code quality was not up to our standards. Also there was no one to maintain these modules, answer user questions, support them.

    So rather than let them rot, we took the hard decision of removing these contributed modules from Gensim. If anyone's interested in maintaining them, please fork & publish into your own repo. They can live happily outside of Gensim.

  • Dropped Python 2. Gensim 4.0 is Py3.6+. Read our Python version support policy.

    • If you still need Python 2 for some reason, stay at Gensim 3.8.3.

  • A new Gensim website – finally! 🙃

So, a major clean-up release overall. We're happy with this tighter, leaner and faster Gensim.

This is the direction we'll keep going forward: less kitchen-sink of "latest academic algorithms", more focus on robust engineering, targetting concrete NLP & document similarity use-cases.

:+1: New features

:books: Tutorials and docs

:red_circle: Bug fixes

  • #2891: Fix fastText word-vectors with ngrams off, by @gojomo
  • #2907: Fix doc2vec crash for large sets of doc-vectors, by @gojomo
  • #2899: Fix similarity bug in NMSLIB indexer, by @piskvorky
  • #2899: Fix deprecation warnings in Annoy integration, by @piskvorky
  • #2901: Fix inheritance of WikiCorpus from TextCorpus, by @jenishah
  • #2940: Fix deprecations in SoftCosineSimilarity, by @Witiko
  • #2944: Fix save_facebook_model failure after update-vocab & other initialization streamlining, by @gojomo
  • #2846: Fix for Python 3.9/3.10: remove xml.etree.cElementTree, by @hugovk
  • #2973: phrases.export_phrases() doesn't yield all bigrams, by @piskvorky
  • #2942: Segfault when training doc2vec, by @gojomo
  • #3041: Fix RuntimeError in export_phrases (change defaultdict to dict), by @thalishsajeed
  • #3059: Fix race condition in FastText tests, by @sleepy-owl

:warning: Removed functionality & deprecations

🔮 Testing, CI, housekeeping

4.0.0.rc1, 2021-03-19

⚠️ Gensim 4.0 contains breaking API changes! See the Migration guide to update your existing Gensim 3.x code and models.

Gensim 4.0 is a major release with lots of performance & robustness improvements and a new website.

Main highlights (see also 👍 Improvements below)

a. Efficiency

| model | 3.8.3: wall time / peak RAM / throughput | 4.0.0: wall time / peak RAM / throughput |
|----------|------------|--------|
| fastText | 2.9h / 4.11 GB / 822k words/s | 2.3h / **1.26 GB** / 914k words/s |
| word2vec | 1.7h / 0.36 GB / 1685k words/s | **1.2h** / 0.33 GB / 1762k words/s |

In other words, fastText now needs 3x less RAM (and is faster); word2vec has 2x faster init (and needs less RAM, and is faster); detecting collocation phrases is 2x faster. ([4.0 benchmarks](https://github.com/RaRe-Technologies/gensim/issues/2887#issuecomment-711097334))

b. Robustness. We fixed a bunch of long-standing bugs by refactoring the internal code structure (see 🔴 Bug fixes below)

c. Simplified OOP model for easier model exports and integration with TensorFlow, PyTorch &co.

These improvements come to you transparently aka "for free", but see Migration guide for some changes that break the old Gensim 3.x API. Update your code accordingly.

  • Dropped a bunch of externally contributed modules: summarization, pivoted TFIDF normalization, FIXME.

    • Code quality was not up to our standards. Also there was no one to maintain them, answer user questions, support these modules.

    So rather than let them rot, we took the hard decision of removing these contributed modules from Gensim. If anyone's interested in maintaining them please fork into your own repo, they can live happily outside of Gensim.

  • Dropped Python 2. Gensim 4.0 is Py3.6+. Read our Python version support policy.

    • If you still need Python 2 for some reason, stay at Gensim 3.8.3.

  • A new Gensim website – finally! 🙃

So, a major clean-up release overall. We're happy with this tighter, leaner and faster Gensim.

This is the direction we'll keep going forward: less kitchen-sink of "latest academic algorithms", more focus on robust engineering, targetting common concrete NLP & document similarity use-cases.

:star2: New Features

:red_circle: Bug fixes

:books: Tutorial and doc improvements

  • fix various documentation warnings (mpenkov, #3077)
  • Fix broken link in rundoc how-to (sezanzeb_, #2991)
  • Point WordEmbeddingSimilarityIndex documentation to gensim.similarities (Witiko, #3003)
  • Make the link to the Gensim 3.8.3 documentation dynamic (Witiko, #2996)

:warning: Removed functionality

🔮 Miscellaneous

4.0.0beta, 2020-10-31

⚠️ Gensim 4.0 contains breaking API changes! See the Migration guide to update your existing Gensim 3.x code and models.

Gensim 4.0 is a major release with lots of performance & robustness improvements and a new website.

Main highlights (see also 👍 Improvements below)

a. Efficiency

| model | 3.8.3: wall time / peak RAM / throughput | 4.0.0: wall time / peak RAM / throughput |
|----------|------------|--------|
| fastText | 2.9h / 4.11 GB / 822k words/s | 2.3h / **1.26 GB** / 914k words/s |
| word2vec | 1.7h / 0.36 GB / 1685k words/s | **1.2h** / 0.33 GB / 1762k words/s |

In other words, fastText now needs 3x less RAM (and is faster); word2vec has 2x faster init (and needs less RAM, and is faster); detecting collocation phrases is 2x faster. ([4.0 benchmarks](https://github.com/RaRe-Technologies/gensim/issues/2887#issuecomment-711097334))

b. Robustness. We fixed a bunch of long-standing bugs by refactoring the internal code structure (see 🔴 Bug fixes below)

c. Simplified OOP model for easier model exports and integration with TensorFlow, PyTorch &co.

These improvements come to you transparently aka "for free", but see Migration guide for some changes that break the old Gensim 3.x API. Update your code accordingly.

  • Dropped a bunch of externally contributed modules: summarization, pivoted TFIDF normalization, FIXME.

    • Code quality was not up to our standards. Also there was no one to maintain them, answer user questions, support these modules.

    So rather than let them rot, we took the hard decision of removing these contributed modules from Gensim. If anyone's interested in maintaining them please fork into your own repo, they can live happily outside of Gensim.

  • Dropped Python 2. Gensim 4.0 is Py3.6+. Read our Python version support policy.

    • If you still need Python 2 for some reason, stay at Gensim 3.8.3.

  • A new Gensim website – finally! 🙃

So, a major clean-up release overall. We're happy with this tighter, leaner and faster Gensim.

This is the direction we'll keep going forward: less kitchen-sink of "latest academic algorithms", more focus on robust engineering, targetting common concrete NLP & document similarity use-cases.

Why pre-release?

This 4.0.0beta pre-release is for users who want the cutting edge performance and bug fixes. Plus users who want to help out, by testing and providing feedback: code, documentation, workflows… Please let us know on the mailing list!

Install the pre-release with:

bash pip install --pre --upgrade gensim

What will change between this pre-release and a "full" 4.0 release?

Production stability is important to Gensim, so we're improving the process of upgrading already-trained saved models. There'll be an explicit model upgrade script between each 4.n to 4.(n+1) Gensim release. Check progress here.

:+1: Improvements

:books: Tutorials and docs

:red_circle: Bug fixes

  • #2891: Fix fastText word-vectors with ngrams off, by @gojomo
  • #2907: Fix doc2vec crash for large sets of doc-vectors, by @gojomo
  • #2899: Fix similarity bug in NMSLIB indexer, by @piskvorky
  • #2899: Fix deprecation warnings in Annoy integration, by @piskvorky
  • #2901: Fix inheritance of WikiCorpus from TextCorpus, by @jenishah
  • #2940; Fix deprecations in SoftCosineSimilarity, by @Witiko
  • #2944: Fix save_facebook_model failure after update-vocab & other initialization streamlining, by @gojomo
  • #2846: Fix for Python 3.9/3.10: remove xml.etree.cElementTree, by @hugovk
  • #2973: phrases.export_phrases() doesn't yield all bigrams
  • #2942: Segfault when training doc2vec

:warning: Removed functionality & deprecations

  • #6: No more binary wheels for x32 platforms, by menshikh-iv
  • #2899: Renamed overly broad similarities.index to the more appropriate similarities.annoy, by @piskvorky
  • #2958: Remove gensim.summarization subpackage, docs and test data, by @mpenkov
  • #2926: Rename num_words to topn in dtm_coherence, by @MeganStodel
  • #2937: Remove Keras dependency, by @piskvorky
  • Removed all code, methods, attributes and functions marked as deprecated in Gensim 3.8.3.
  • Removed pattern dependency (PR #3012, @mpenkov). If you need to lemmatize, do it prior to passing the corpus to gensim.

- Python
Published by piskvorky almost 5 years ago

gensim -

4.0.1, 2021-04-01

Bugfix release to address issues with wheels on Windows due to Numpy binary incompatibility:

  • https://github.com/RaRe-Technologies/gensim/issues/3095
  • https://github.com/RaRe-Technologies/gensim/issues/3097

4.0.0, 2021-03-24

⚠️ Gensim 4.0 contains breaking API changes! See the Migration guide to update your existing Gensim 3.x code and models.

Gensim 4.0 is a major release with lots of performance & robustness improvements, and a new website.

Main highlights

a. Efficiency

| model | 3.8.3: wall time / peak RAM / throughput | 4.0.0: wall time / peak RAM / throughput |
|----------|------------|--------|
| fastText | 2.9h / 4.11 GB / 822k words/s | 2.3h / **1.26 GB** / 914k words/s |
| word2vec | 1.7h / 0.36 GB / 1685k words/s | **1.2h** / 0.33 GB / 1762k words/s |

In other words, fastText now needs 3x less RAM (and is faster); word2vec has 2x faster init (and needs less RAM, and is faster); detecting collocation phrases is 2x faster. ([4.0 benchmarks](https://github.com/RaRe-Technologies/gensim/issues/2887#issuecomment-711097334))

b. Robustness. We fixed a bunch of long-standing bugs by refactoring the internal code structure (see 🔴 Bug fixes below)

c. Simplified OOP model for easier model exports and integration with TensorFlow, PyTorch &co.

These improvements come to you transparently aka "for free", but see Migration guide for some changes that break the old Gensim 3.x API. Update your code accordingly.

  • Dropped a bunch of externally contributed modules and wrappers: summarization, pivoted TFIDF, Mallet…

    • Code quality was not up to our standards. Also there was no one to maintain these modules, answer user questions, support them.

    So rather than let them rot, we took the hard decision of removing these contributed modules from Gensim. If anyone's interested in maintaining them, please fork & publish into your own repo. They can live happily outside of Gensim.

  • Dropped Python 2. Gensim 4.0 is Py3.6+. Read our Python version support policy.

    • If you still need Python 2 for some reason, stay at Gensim 3.8.3.

  • A new Gensim website – finally! 🙃

So, a major clean-up release overall. We're happy with this tighter, leaner and faster Gensim.

This is the direction we'll keep going forward: less kitchen-sink of "latest academic algorithms", more focus on robust engineering, targetting concrete NLP & document similarity use-cases.

:+1: New features

:books: Tutorials and docs

:red_circle: Bug fixes

  • #2891: Fix fastText word-vectors with ngrams off, by @gojomo
  • #2907: Fix doc2vec crash for large sets of doc-vectors, by @gojomo
  • #2899: Fix similarity bug in NMSLIB indexer, by @piskvorky
  • #2899: Fix deprecation warnings in Annoy integration, by @piskvorky
  • #2901: Fix inheritance of WikiCorpus from TextCorpus, by @jenishah
  • #2940: Fix deprecations in SoftCosineSimilarity, by @Witiko
  • #2944: Fix save_facebook_model failure after update-vocab & other initialization streamlining, by @gojomo
  • #2846: Fix for Python 3.9/3.10: remove xml.etree.cElementTree, by @hugovk
  • #2973: phrases.export_phrases() doesn't yield all bigrams, by @piskvorky
  • #2942: Segfault when training doc2vec, by @gojomo
  • #3041: Fix RuntimeError in export_phrases (change defaultdict to dict), by @thalishsajeed
  • #3059: Fix race condition in FastText tests, by @sleepy-owl

:warning: Removed functionality & deprecations

🔮 Testing, CI, housekeeping

4.0.0.rc1, 2021-03-19

⚠️ Gensim 4.0 contains breaking API changes! See the Migration guide to update your existing Gensim 3.x code and models.

Gensim 4.0 is a major release with lots of performance & robustness improvements and a new website.

Main highlights (see also 👍 Improvements below)

a. Efficiency

| model | 3.8.3: wall time / peak RAM / throughput | 4.0.0: wall time / peak RAM / throughput |
|----------|------------|--------|
| fastText | 2.9h / 4.11 GB / 822k words/s | 2.3h / **1.26 GB** / 914k words/s |
| word2vec | 1.7h / 0.36 GB / 1685k words/s | **1.2h** / 0.33 GB / 1762k words/s |

In other words, fastText now needs 3x less RAM (and is faster); word2vec has 2x faster init (and needs less RAM, and is faster); detecting collocation phrases is 2x faster. ([4.0 benchmarks](https://github.com/RaRe-Technologies/gensim/issues/2887#issuecomment-711097334))

b. Robustness. We fixed a bunch of long-standing bugs by refactoring the internal code structure (see 🔴 Bug fixes below)

c. Simplified OOP model for easier model exports and integration with TensorFlow, PyTorch &co.

These improvements come to you transparently aka "for free", but see Migration guide for some changes that break the old Gensim 3.x API. Update your code accordingly.

  • Dropped a bunch of externally contributed modules: summarization, pivoted TFIDF normalization, FIXME.

    • Code quality was not up to our standards. Also there was no one to maintain them, answer user questions, support these modules.

    So rather than let them rot, we took the hard decision of removing these contributed modules from Gensim. If anyone's interested in maintaining them please fork into your own repo, they can live happily outside of Gensim.

  • Dropped Python 2. Gensim 4.0 is Py3.6+. Read our Python version support policy.

    • If you still need Python 2 for some reason, stay at Gensim 3.8.3.

  • A new Gensim website – finally! 🙃

So, a major clean-up release overall. We're happy with this tighter, leaner and faster Gensim.

This is the direction we'll keep going forward: less kitchen-sink of "latest academic algorithms", more focus on robust engineering, targetting common concrete NLP & document similarity use-cases.

:star2: New Features

:red_circle: Bug fixes

:books: Tutorial and doc improvements

  • fix various documentation warnings (mpenkov, #3077)
  • Fix broken link in rundoc how-to (sezanzeb_, #2991)
  • Point WordEmbeddingSimilarityIndex documentation to gensim.similarities (Witiko, #3003)
  • Make the link to the Gensim 3.8.3 documentation dynamic (Witiko, #2996)

:warning: Removed functionality

🔮 Miscellaneous

4.0.0beta, 2020-10-31

⚠️ Gensim 4.0 contains breaking API changes! See the Migration guide to update your existing Gensim 3.x code and models.

Gensim 4.0 is a major release with lots of performance & robustness improvements and a new website.

Main highlights (see also 👍 Improvements below)

a. Efficiency

| model | 3.8.3: wall time / peak RAM / throughput | 4.0.0: wall time / peak RAM / throughput |
|----------|------------|--------|
| fastText | 2.9h / 4.11 GB / 822k words/s | 2.3h / **1.26 GB** / 914k words/s |
| word2vec | 1.7h / 0.36 GB / 1685k words/s | **1.2h** / 0.33 GB / 1762k words/s |

In other words, fastText now needs 3x less RAM (and is faster); word2vec has 2x faster init (and needs less RAM, and is faster); detecting collocation phrases is 2x faster. ([4.0 benchmarks](https://github.com/RaRe-Technologies/gensim/issues/2887#issuecomment-711097334))

b. Robustness. We fixed a bunch of long-standing bugs by refactoring the internal code structure (see 🔴 Bug fixes below)

c. Simplified OOP model for easier model exports and integration with TensorFlow, PyTorch &co.

These improvements come to you transparently aka "for free", but see Migration guide for some changes that break the old Gensim 3.x API. Update your code accordingly.

  • Dropped a bunch of externally contributed modules: summarization, pivoted TFIDF normalization, FIXME.

    • Code quality was not up to our standards. Also there was no one to maintain them, answer user questions, support these modules.

    So rather than let them rot, we took the hard decision of removing these contributed modules from Gensim. If anyone's interested in maintaining them please fork into your own repo, they can live happily outside of Gensim.

  • Dropped Python 2. Gensim 4.0 is Py3.6+. Read our Python version support policy.

    • If you still need Python 2 for some reason, stay at Gensim 3.8.3.

  • A new Gensim website – finally! 🙃

So, a major clean-up release overall. We're happy with this tighter, leaner and faster Gensim.

This is the direction we'll keep going forward: less kitchen-sink of "latest academic algorithms", more focus on robust engineering, targetting common concrete NLP & document similarity use-cases.

Why pre-release?

This 4.0.0beta pre-release is for users who want the cutting edge performance and bug fixes. Plus users who want to help out, by testing and providing feedback: code, documentation, workflows… Please let us know on the mailing list!

Install the pre-release with:

bash pip install --pre --upgrade gensim

What will change between this pre-release and a "full" 4.0 release?

Production stability is important to Gensim, so we're improving the process of upgrading already-trained saved models. There'll be an explicit model upgrade script between each 4.n to 4.(n+1) Gensim release. Check progress here.

:+1: Improvements

:books: Tutorials and docs

:red_circle: Bug fixes

  • #2891: Fix fastText word-vectors with ngrams off, by @gojomo
  • #2907: Fix doc2vec crash for large sets of doc-vectors, by @gojomo
  • #2899: Fix similarity bug in NMSLIB indexer, by @piskvorky
  • #2899: Fix deprecation warnings in Annoy integration, by @piskvorky
  • #2901: Fix inheritance of WikiCorpus from TextCorpus, by @jenishah
  • #2940; Fix deprecations in SoftCosineSimilarity, by @Witiko
  • #2944: Fix save_facebook_model failure after update-vocab & other initialization streamlining, by @gojomo
  • #2846: Fix for Python 3.9/3.10: remove xml.etree.cElementTree, by @hugovk
  • #2973: phrases.export_phrases() doesn't yield all bigrams
  • #2942: Segfault when training doc2vec

:warning: Removed functionality & deprecations

  • #6: No more binary wheels for x32 platforms, by menshikh-iv
  • #2899: Renamed overly broad similarities.index to the more appropriate similarities.annoy, by @piskvorky
  • #2958: Remove gensim.summarization subpackage, docs and test data, by @mpenkov
  • #2926: Rename num_words to topn in dtm_coherence, by @MeganStodel
  • #2937: Remove Keras dependency, by @piskvorky
  • Removed all code, methods, attributes and functions marked as deprecated in Gensim 3.8.3.
  • Removed pattern dependency (PR #3012, @mpenkov). If you need to lemmatize, do it prior to passing the corpus to gensim.

3.8.3, 2020-05-03

:warning: 3.8.x will be the last Gensim version to support Py2.7. Starting with 4.0.0, Gensim will only support Py3.5 and above.

This is primarily a bugfix release to bring back Py2.7 compatibility to gensim 3.8.

:red_circle: Bug fixes

  • Bring back Py27 support (PR #2812, @mpenkov)
  • Fix wrong version reported by setup.py (Issue #2796)
  • Fix missing C extensions (Issues #2794 and #2802)

:+1: Improvements

:books: Tutorial and doc improvements

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use loadfacebookvectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and loadfacebookmodel to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

3.8.2, 2020-04-10

:red_circle: Bug fixes

  • Pin smart_open version for compatibility with Py2.7

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use loadfacebookvectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and loadfacebookmodel to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

3.8.1, 2019-09-23

:red_circle: Bug fixes

  • Fix usage of basedir instead of BASEDIR in loadinfo in downloader. (movb, #2605)
  • Update the version of smart_open in the setup.py file (AMR-KELEG, #2582)
  • Properly handle unicode_errors arg parameter when loading a vocab file (wmtzk, #2570)
  • Catch loading older TfidfModels without smartirs (bnomis, #2559)
  • Fix bug where a module import set up logging, pin doctools for Py2 (piskvorky, #2552)

:books: Tutorial and doc improvements

:+1: Improvements

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use loadfacebookvectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and loadfacebookmodel to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

3.8.0, 2019-07-08

:star2: New Features

  • Enable online training of Poincare models (koiizukag, #2505)
  • Make BM25 more scalable by adding support for generator inputs (saraswatmks, #2479)
  • Allow the Gensim dataset / pre-trained model downloader gensim.downloader to run offline, by introducing a local file cache (mpenkov, #2545)
  • Make the gensim.downloader target directory configurable (mpenkov, #2456)
  • Add nmslib indexer (masa3141, #2417)

:red_circle: Bug fixes

  • Fix smart_open deprecation warning globally (itayB, #2530)
  • Fix AppVeyor issues with Windows and Py2 (mpenkov, #2546)
  • Fix topn=0 versus topn=None bug in most_similar, accept topn of any integer type (Witiko, #2497)
  • Fix Python version check (charsyam, #2547)
  • Fix typo in FastText documentation (Guitaricet, #2518)
  • Fix "Market Matrix" to "Matrix Market" typo. (Shooter23, #2513)
  • Fix auto-generated hyperlinks in CHANGELOG.md (mpenkov, #2482)

:books: Tutorial and doc improvements

  • Generate documentation for the gensim.similarities.termsim module (Witiko, #2485)
  • Simplify the Support section in README (piskvorky, #2542)

:+1: Improvements

  • Pin sklearn version for Py2, because sklearn dropped py2 support (mpenkov, #2510)

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use loadfacebookvectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and loadfacebookmodel to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

3.7.3, 2019-05-06

:red_circle: Bug fixes

:books: Tutorial and doc improvements

:+1: Improvements

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use loadfacebookvectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and loadfacebookmodel to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

3.7.2, 2019-04-06

:star2: New Features

  • gensim.models.fasttext.load_facebook_model function: load full model (slower, more CPU/memory intensive, supports training continuation)
  • gensim.models.fasttext.load_facebook_vectors function: load embeddings only (faster, less CPU/memory usage, does not support training continuation)

:red_circle: Bug fixes

  • Fix unicode error when loading FastText vocabulary (@mpenkov, #2390)
  • Avoid division by zero in fasttext_inner.pyx (@mpenkov, #2404)
  • Avoid incorrect filename inference when loading model (@mpenkov, #2408)
  • Handle invalid unicode when loading native FastText models (@mpenkov, #2411)
  • Avoid divide by zero when calculating vectors for terms with no ngrams (@mpenkov, #2411)

:books: Tutorial and doc improvements

:+1: Improvements

:warning: Changes in FastText behavior

Out-of-vocab word handling

To achieve consistency with the reference implementation from Facebook, a FastText model will now always report any word, out-of-vocabulary or not, as being in the model, and always return some vector for any word looked-up. Specifically:

  1. 'any_word' in ft_model will always return True. Previously, it returned True only if the full word was in the vocabulary. (To test if a full word is in the known vocabulary, you can consult the wv.vocab property: 'any_word' in ft_model.wv.vocab will return False if the full word wasn't learned during model training.)
  2. ft_model['any_word'] will always return a vector. Previously, it raised KeyError for OOV words when the model had no vectors for any ngrams of the word.
  3. If no ngrams from the term are present in the model, or when no ngrams could be extracted from the term, a vector pointing to the origin will be returned. Previously, a vector of NaN (not a number) was returned as a consequence of a divide-by-zero problem.
  4. Models may use more more memory, or take longer for word-vector lookup, especially after training on smaller corpuses where the previous non-compliant behavior discarded some ngrams from consideration.

Loading models in Facebook .bin format

The gensim.models.FastText.load_fasttext_format function (deprecated) now loads the entire model contained in the .bin file, including the shallow neural network that enables training continuation. Loading this NN requires more CPU and RAM than previously required.

Since this function is deprecated, consider using one of its alternatives (see below).

Furthermore, you must now pass the full path to the file to load, including the file extension. Previously, if you specified a model path that ends with anything other than .bin, the code automatically appended .bin to the path before loading the model. This behavior was confusing, so we removed it.

:warning: Deprecations (will be removed in the next major release)

Remove:

  • gensim.models.FastText.load_fasttext_format: use loadfacebookvectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and loadfacebookmodel to load full model (slower, more CPU/memory intensive, supports training continuation)

3.7.1, 2019-01-31

:+1: Improvements

:red_circle: Bug fixes

  • Fix infinite diff in LdaModel.do_mstep (@horpto, #2344)
  • Fix backward compatibility issue: loading FastTextKeyedVectors using KeyedVectors (missing attribute compatible_hash) (@menshikh-iv, #2349)
  • Fix logging issue (conda-forge related) (@menshikh-iv, #2339)
  • Fix WordEmbeddingsKeyedVectors.most_similar (@Witiko, #2356)
  • Fix issues of flake8==3.7.1 (@horpto, #2365)

:books: Tutorial and doc improvements

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

3.7.0, 2019-01-18

:star2: New features

  • Fast Online NMF (@anotherbugmaster, #2007)

    • Benchmark wiki-english-20171001

    | Model | Perplexity | Coherence | L2 norm | Train time (minutes) | |-------|------------|-----------|---------|----------------------| | LDA | 4727.07 | -2.514 | 7.372 | 138 | | NMF | 975.74 | -2.814 | 7.265 | 73 | | NMF (with regularization) | 985.57 | -2.436 | 7.269 | 441 | - Simple to use (same interface as LdaModel) ```python from gensim.models.nmf import Nmf from gensim.corpora import Dictionary import gensim.downloader as api

    text8 = api.load('text8')

    dictionary = Dictionary(text8) dictionary.filter_extremes()

    corpus = [ dictionary.doc2bow(doc) for doc in text8 ]

    nmf = Nmf( corpus=corpus, numtopics=5, id2word=dictionary, chunksize=2000, passes=5, randomstate=42, )

    nmf.show_topics() """ [(0, '0.007"km" + 0.006"est" + 0.006"islands" + 0.004"league" + 0.004"rate" + 0.004"female" + 0.004"economy" + 0.003"male" + 0.003"team" + 0.003"elections"'), (1, '0.006"actor" + 0.006"player" + 0.004"bwv" + 0.004"writer" + 0.004"actress" + 0.004"singer" + 0.003"emperor" + 0.003"jewish" + 0.003"italian" + 0.003"prize"'), (2, '0.036"college" + 0.007"institute" + 0.004"jewish" + 0.004"universidad" + 0.003"engineering" + 0.003"colleges" + 0.003"connecticut" + 0.003"technical" + 0.003"jews" + 0.003"universities"'), (3, '0.016"import" + 0.008"insubstantial" + 0.007"y" + 0.006"soviet" + 0.004"energy" + 0.004"info" + 0.003"duplicate" + 0.003"function" + 0.003"z" + 0.003"jargon"'), (4, '0.005"software" + 0.004"games" + 0.004"windows" + 0.003"microsoft" + 0.003"films" + 0.003"apple" + 0.003"video" + 0.002"album" + 0.002"fiction" + 0.002"characters"')] """ ``` - See also: - NMF tutorial - Full NMF Benchmark

  • Massive improvementFastText compatibilities (@mpenkov, #2313) ```python from gensim.models import FastText

    'cc.ru.300.bin' - Russian Facebook FT model trained on Common Crawl

    Can be downloaded from https://s3-us-west-1.amazonaws.com/fasttext-vectors/word-vectors-v2/cc.ru.300.bin.gz

    model = FastText.loadfasttextformat("cc.ru.300.bin")

    Fixed hash-function allow to produce same output as FB FastText & works correctly for non-latin languages (for example, Russian)

    assert "мяу" in m.wv.vocab # 'мяу' - vocab word model.wv.most_similar("мяу") """ [('Мяу', 0.6820122003555298), ('МЯУ', 0.6373013257980347), ('мяу-мяу', 0.593108594417572), ('кис-кис', 0.5899622440338135), ('гав', 0.5866007804870605), ('Кис-кис', 0.5798211097717285), ('Кис-кис-кис', 0.5742273330688477), ('Мяу-мяу', 0.5699705481529236), ('хрю-хрю', 0.5508339405059814), ('ав-ав', 0.5479759573936462)] """

    assert "котогород" not in m.wv.vocab # 'котогород' - out-of-vocab word model.wv.most_similar("котогород", topn=3) """ [('автогород', 0.5463314652442932), ('ТагилНовокузнецкНовомосковскНовороссийскНовосибирскНовотроицкНовочеркасскНовошахтинскНовый', 0.5423436164855957), ('областьНовосибирскБарабинскБердскБолотноеИскитимКарасукКаргатКуйбышевКупиноОбьТатарскТогучинЧерепаново', 0.5377570390701294)] """

    Now we load full model, for this reason, we can continue an training

    from gensim.test.utils import datapath from smartopen import smartopen

    with smart_open(datapath("crime-and-punishment.txt"), encoding="utf-8") as infile: # russian text corpus = [line.strip().split() for line in infile]

    model.train(corpus, total_examples=len(corpus), epochs=5) ```

  • Similarity search improvements (@Witiko, #2016)

    • Add similarity search using the Levenshtein distance in gensim.similarities.LevenshteinSimilarityIndex
    • Performance optimizations to gensim.similarities.SoftCosineSimilarity (full benchmark)

    | dictionary size | corpus size | speed | |-----------------|-------------|--------------:| | 1000 | 100 | 1.0× | | 1000 | 1000 | 53.4× | | 1000 | 100000 | 156784.8× | | 100000 | 100 | 3.8× | | 100000 | 1000 | 405.8× | | 100000 | 100000 | 66262.0× | - See updated soft-cosine tutorial for more information and usage examples

  • Add python3.7 support (@menshikh-iv, #2211)

:+1: Improvements

Optimizations
  • Reduce Phraser memory usage (drop frequencies) (@jenishah, #2208)
  • Reduce memory consumption of summarizer (@horpto, #2298)
  • Replace inline slow equivalent of meanabsolutedifference with fast (@horpto, #2284)
  • Reuse precalculated updated prior in ldamodel.update_dir_prior (@horpto, #2274)
  • Improve KeyedVector.wmdistance (@horpto, #2326)
  • Optimize remove_unreachable_nodes in gensim.summarization (@horpto, #2263)
  • Optimize mz_entropy from gensim.summarization (@horpto, #2267)
  • Improve filter_extremes methods in Dictionary and HashDictionary (@horpto, #2303)
Additions
Cleanup

:red_circle: Bug fixes

:books: Tutorial and doc improvements

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

3.6.0, 2018-09-20

:star2: New features

New training mode for *2Vec models (word2vec, doc2vec, fasttext) that allows model training to scale linearly with the number of cores (full GIL elimination). The result of our Google Summer of Code 2018 project by Dmitry Persiyanov.

Benchmark - Dataset: full English Wikipedia - Cloud: GCE - CPU: Intel(R) Xeon(R) CPU @ 2.30GHz 32 cores - BLAS: MKL

| Model | Queue-based version [sec] | File-based version [sec] | speed up | Accuracy (queue-based) | Accuracy (file-based) | |-------|------------|--------------------|----------|----------------|-----------------------| | Word2Vec | 9230 | 2437 | 3.79x | 0.754 (Âą 0.003) | 0.750 (Âą 0.001) | | Doc2Vec | 18264 | 2889 | 6.32x | 0.721 (Âą 0.002) | 0.683 (Âą 0.003) | | FastText | 16361 | 10625 | 1.54x | 0.642 (Âą 0.002) | 0.660 (Âą 0.001) |

Usage:

```python import gensim.downloader as api from multiprocessing import cpucount from gensim.utils import saveaslinesentence from gensim.test.utils import get_tmpfile from gensim.models import Word2Vec, Doc2Vec, FastText

# Convert any corpus to the needed format: 1 document per line, words delimited by " " corpus = api.load("text8") corpusfname = gettmpfile("text8-file-sentence.txt") saveaslinesentence(corpus, corpusfname)

# Choose num of cores that you want to use (let's use all, models scale linearly now!) numcores = cpucount()

# Train models using all cores w2vmodel = Word2Vec(corpusfile=corpusfname, workers=numcores) d2vmodel = Doc2Vec(corpusfile=corpusfname, workers=numcores) ftmodel = FastText(corpusfile=corpusfname, workers=numcores)

``` Read notebook tutorial with full description.

:+1: Improvements

:red_circle: Bug fixes

:books: Tutorial and doc improvements

  • Update docstring with new analogy evaluation method (@akutuzov, #2130)
  • Improve prune_at parameter description for gensim.corpora.Dictionary (@yxonic, #2128)
  • Fix default -> auto prior parameter in documentation for lda-related models (@Laubeee, #2156)
  • Use heading instead of bold style in gensim.models.translation_matrix (@nzw0301, #2164)
  • Fix quote of vocabulary from gensim.models.Word2Vec (@nzw0301, #2161)
  • Replace deprecated parameters with new in docstring of gensim.models.Doc2Vec (@xuhdev, #2165)
  • Fix formula in Mallet documentation (@Laubeee, #2186)
  • Fix minor semantic issue in docs for Phrases (@RunHorst, #2148)
  • Fix typo in documentation (@KenjiOhtsuka, #2157)
  • Additional documentation fixes (@piskvorky, #2121)

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

3.5.0, 2018-07-06

This release comprises a glorious 38 pull requests from 28 contributors. Most of the effort went into improving the documentation—hence the release code name "Docs 💬"!

Apart from the massive overhaul of all Gensim documentation (including docstring style and examples—you asked for it), we also managed to sneak in some new functionality and a number of bug fixes. As usual, see the notes below for a complete list, with links to pull requests for more details.

Huge thanks to all contributors! Nobody loves working on documentation. 3.5.0 is a result of several months of laborious, unglamorous, and sometimes invisible work. Enjoy!

:books: Documentation improvements

:star2: New features:

  • Add add_entity method to KeyedVectors to allow adding word vectors manually (@persiyanov, #1957)
  • Add inference for new unseen author to AuthorTopicModel (@Stamenov, #1766)
  • Add evaluate_word_analogies (will replace accuracy) method to KeyedVectors (@akutuzov, #1935)
  • Add Pivot Normalization to TfidfModel (@markroxor, #1780)

:+1: Improvements

:red_circle: Bug fixes:

  • Fix Doc2Vec.infer_vector + notebook cleanup (@gojomo, #2103)
  • Fix linear decay for learning rate in Doc2Vec.infer_vector (@umangv, #2063)
  • Fix negative sampling floating-point error for `gensim.models.Poincare (@jayantj, #1959)
  • Fix loading word2vec and doc2vec models saved using old Gensim versions (@manneshiva, #2012)
  • Fix SoftCosineSimilarity.get_similarities on corpora ssues/1955) (@Witiko, #1972)
  • Fix return dtype for matutils.unitvec according to input dtype (@o-P-o, #1992)
  • Fix passing empty dictionary to gensim.corpora.WikiCorpus (@steremma, #2042)
  • Fix bug in Similarity.query_shards in multiprocessing case (@bohea, #2044)
  • Fix SMART from TfidfModel for case when df == "n" (@PeteBleackley, #2021)
  • Fix OverflowError when loading a large term-document matrix in compiled MatrixMarket format (@arlenk, #2001)
  • Update rules for removing table markup from Wikipedia dumps (@chaitaliSaini, #1954)
  • Fix _is_single from Phrases for case when corpus is a NumPy array (@rmalouf, #1987)
  • Fix tests for EuclideanKeyedVectors.similarity_matrix (@Witiko, #1984)
  • Fix deprecated parameters in D2VTransformer and W2VTransformer(@MritunjayMohitesh, #1945)
  • Fix Doc2Vec.infer_vector after loading old Doc2Vec (gensim<=3.2)(@manneshiva, #1974)
  • Fix inheritance chain for load_word2vec_format (@DennisChen0307, #1968)
  • Update Keras version (avoid bug from keras==2.1.5) (@menshikh-iv, #1963)

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

3.4.0, 2018-03-01

:star2: New features:

  • Massive optimizations of gensim.models.LdaModel: much faster training, using Cython. (@arlenk, #1767)

    • Training benchmark :boom:

    | dataset | old LDA [sec] | optimized LDA [sec] | speed up | |---------|---------------|---------------------|---------| | nytimes | 3473 | 1975 | 1.76x | | enron | 774 | 437 | 1.77x | - This change affects all models that depend on LdaModel, such as LdaMulticore, LdaSeqModel, AuthorTopicModel.

  • Huge speed-ups to corpus I/O with MmCorpus (Cython) (@arlenk, #1825)

    • File reading benchmark

    | dataset | file compressed? | old MmReader [sec] | optimized MmReader [sec] | speed up | |---------------|:-----------:|:------------:|:------------------:|:-------------:| | enron | no | 22.3 | 2.6 | 8.7x | | | yes | 37.3 | 14.4 | 2.6x | | nytimes | no | 419.3 | 49.2 | 8.5x | | | yes | 686.2 | 275.1 | 2.5x | | text8 | no | 25.4 | 2.5 | 10.1x | | | yes | 41.9 | 17.0 | 2.5x | - Overall, a 2.5x speedup for compressed .mm.gz input and 8.5x :fire::fire::fire: for uncompressed plaintext .mm.

  • Performance and memory optimization to gensim.models.FastText :rocket: (@jbaiter, #1916)

    • Benchmark (first 500,000 articles from English Wikipedia)

    | Metric | old FastText | optimized FastText | improvement | | -----------------------| -----------------| -------------------|-------------| | Training time (1 epoch) | 4823.4s (80.38 minutes) | 1873.6s (31.22 minutes) | 2.57x | | Training time (full) | 1h 26min 13s | 36min 43s | 2.35x | | Training words/sec | 72,781 | 187,366 | 2.57x | | Training peak memory | 5.2 GB | 3.7 GB | 1.4x | - Overall, a 2.5x speedup & memory usage reduced by 30%.

  • Implemented Soft Cosine Measure (@Witiko, #1827)

    | Technique | MAP score | Duration | |-----------|-----------|--------------| | softcossim| 45.99 | 1.24 sec | | wmd-relax | 44.48 | 12.22 sec | | cossim | 44.22 | 4.39 sec | | wmd-gensim| 44.08 | 98.29 sec | - Soft Cosine notebook with detailed description, examples & benchmarks - Related papers: - Soft Similarity and Soft Cosine Measure: Similarity of Features in Vector Space Model - SimBow at SemEval-2017 Task 3: Soft-Cosine Semantic Similarity between Questions for Community Question Answering - Vector Space Representations in IR

:+1: Improvements:

  • New method to show the Gensim installation parameters: python -m gensim.scripts.package_info --info. Use this when reporting problems, for easier debugging. Fix #1902 (@sharanry, #1903)
  • Added a flag to optionally skip network-related tests, to help maintainers avoid network issues with CI services (@menshikh-iv, #1930)
  • Added license field to setup.py, allowing the use of tools like pip-licenses (@nils-werner, #1909)

:red_circle: Bug fixes:

  • Fix Python 3 compatibility for gensim.corpora.UciCorpus.save_corpus (@darindf, #1875)
  • Add wv property to KeyedVectors for backward compatibility. Fix #1882 (@manneshiva, #1884)
  • Fix deprecation warning from inspect.getargspec. Fix #1878 (@aneesh-joshi, #1887)
  • Add LabeledSentence to gensim.models.doc2vec for backward compatibility. Fix #1886 (@manneshiva, #1891)
  • Fix empty output bug in Phrases (when using model[tokens] twice). Fix #1401 (@sj29-innovate, #1853)
  • Fix type problems for D2VTransformer.fit_transform. Fix #1834 (@Utkarsh-Mishra-CIC, #1845)
  • Fix datatype parameter for KeyedVectors.load_word2vec_format. Fix #1682 (@pushpankar, #1819)
  • Fix deprecated parameters in doc2vec-lee notebook (@TheFlash10, #1918)
  • Fix file-like closing bug in gensim.corpora.MmCorpus. Fix #1869 (@sj29-innovate, #1911)
  • Fix precision problem in test_similarities.py, no more FP fails. (@menshikh-iv, #1928)
  • Fix encoding in Lee corpus reader. (@menshikh-iv, #1931)
  • Fix OOV pairs counter in WordEmbeddingsKeyedVectors.evaluate_word_pairs. (@akutuzov, #1934)

:books: Tutorial and doc improvements:

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

3.3.0, 2018-02-02

:star2: New features: * Re-designed all "*2vec" implementations (@manneshiva, #1777) - Modular organization of Word2Vec, Doc2Vec, FastText, etc ..., making it easier to add new models in the future and re-use code - Fully backward compatible (even with loading models stored by a previous Gensim version) - Detailed documentation for the *2vec refactoring project

  • Improve gensim.scripts.segment_wiki by retaining interwiki links. Fix #1712 (@steremma, PR #1839)

    • Optionally extract interlinks from Wikipedia pages (use the --include-interlinks option). This will output one additional JSON dict for each article: { "interlinks": { "article title 1": "interlink text 1", "article title 2": "interlink text 2", ... } }

    • Example: extract the Wikipedia graph with article links as edges, from a raw Wikipedia dump: bash python -m gensim.scripts.segment_wiki --include-interlinks --file ~/Downloads/enwiki-latest-pages-articles.xml.bz2 --output ~/Desktop/enwiki-latest.jsonl.gz

      • Read this field from the segment_wiki output:

      ```python import json from smartopen import smartopen

      with smart_open("enwiki-latest.jsonl.gz") as infile: for doc in infile: doc = json.loads(doc)

          src_node = doc['title']
    dst_nodes = doc['interlinks'].keys()

    print(u"Source node: {}".format(src_node))
    print(u"Destination nodes: {}".format(u", ".join(dst_nodes)))
    break

      """ OUTPUT:

      Source node: Anarchism Destination nodes: anarcha-feminist, Ivan Illich, Adolf Brand, Josiah Warren, will (philosophy), anarcha-feminism, Anarchism in Mexico, Lysander Spooner, English Civil War, G8, Sebastien Faure, Nihilist movement, Sébastien Faure, Left-wing politics, imamate, Pierre Joseph Proudhon, anarchist communism, Università popolare (Italian newspaper), 1848 Revolution, Synthesis anarchism, labour movement, anarchist communists, collectivist anarchism, polyamory, post-humanism, postcolonialism, anti war movement, State (polity), security culture, Catalan people, Stoicism, Progressive education, stateless society, Umberto I of Italy, German language, Anarchist schools of thought, NEFAC, Jacques Ellul, Spanish Communist Party, Crypto-anarchism, ruling class, non-violence, Platformist, The History of Sexuality, Revolutions of 1917–23, Federación Anarquista Ibérica, propaganda of the deed, William B. Greene, Platformism, mutually exclusive, Fraye Arbeter Shtime, Adolf Hitler, oxymoron, Paris Commune, Anarchism in Italy#Postwar years and today, Oranienburg, abstentionism, Free Society, Henry David Thoreau, privative alpha, George I of Greece, communards, Gustav Landauer, Lucifer the Lightbearer, Moses Harman, coercion, regicide, rationalist, Resistance during World War II, Christ (title), Bohemianism, individualism, Crass, black bloc, Spanish Revolution of 1936, Erich Mühsam, Empress Elisabeth of Austria, Free association (communism and anarchism), general strike, Francesc Ferrer i Guàrdia, Catalan anarchist pedagogue and free-thinker, veganarchism, Traditional knowledge, Japanese Anarchist Federation, Diogenes of Sinope, Hierarchy, sexual revolution, Naturism, Bavarian Soviet Republic, February Revolution, Eugene Varlin, Renaissance humanism, Mexican Liberal Party, Friedrich Engels, Fernando Tarrida del Mármol, Caliphate, Marxism, Jesus, John Cage, Umanita Nova, Anarcho-pacifism, Peter Kropotkin, Religious anarchism, Anselme Bellegarrigue, civilisation, moral obligation, hedonist, Free Territory (Ukraine), -ism, neo-liberalism, Austrian School, philosophy, freethought, Joseph Goebbels, Conservatism, anarchist economics, Cavalier, Maximilien de Robespierre, Comstockery, Dorothy Day, Anarchism in France, Fédération anarchiste, World Economic Forum, Amparo Poch y Gascón, Sex Pistols, women's rights, collectivisation, Taoism, common ownership, William Batchelder Greene, Collective farming, popular education, biphobia, targeted killings, Protestant Christianity, state socialism, Marie François Sadi Carnot, Stephen Pearl Andrews, World Trade Organization, Communist Party of Spain (main), Pluto Press, Levante, Spain, Alexander Berkman, Wilhelm Weitling, Kharijites, Bolshevik, Liberty (1881–1908), Anarchist Aragon, social democrats, Dielo Truda, Post-left anarchy, Age of Enlightenment, Blanquism, Walden, mutual aid (organization), Far-left politics, privative, revolutions of 1848, anarchism and nationalism, punk rock, Étienne de La Boétie, Max Stirner, Jacobin (politics), agriculture, anarchy, Confederacion General del Trabajo de España, toleration, reformism, International Anarchist Congress of Amsterdam, The Ego and Its Own, Ukraine, Civil Disobedience (Thoreau), Spanish Civil War, David Graeber, Anarchism and issues related to love and sex, James Guillaume, Insurrectionary anarchism, Political repression, International Workers' Association, Barcelona, Bulgaria, Voline, Zeno of Citium, anarcho-communists, organized religion, libertarianism, bisexuality, Ricardo Flores Magón, Henri Zisly, Eight-hour day, Freetown Christiania, heteronormativity, Mikhail Bakunin, Propagandaministerium, Ezra Heywood, individual reappropriation, Modern School (United States), archon, Confédération nationale du travail, socialist movement, History of Islam, Max Nettlau, Political Justice, Reichstag fire, Anti-Christianity, decentralised, Issues in anarchism#Communism, deschooling, Christian movement, squatter, Anarchism in Germany, Catalonia, Louise Michel, Solidarity Federation, What is Property?, European individualist anarchism, Pierre-Joseph Proudhon, Mexican Revolution, wikt:anarchism, Blackshirts, Jewish anarchism, Russian Civil War, property rights, anti-authoritarian, individual reclamation, propaganda by the deed, from each according to his ability, to each according to his need, Feminist movement, Confiscation, social anarchism, Anarchism in Russia, Daniel Guérin, Uruguayan Anarchist Federation, Anarcha-feminism, Enragés, Cynicism (philosophy), workers' council, The Word (free love), Allen Ginsberg, Campaign for Nuclear Disarmament, antimilitarism, Workers' self-management, Federación Obrera Regional Argentina, self-governance, free market, Carlos I of Portugal, Simon Critchley, Anti-clericalism, heterosexual, Layla AbdelRahim, Mexican Anarchist Federation, Anarchism and Marxism, October Revolution, Anti-nuclear movement, Joseph Déjacque, Bolsheviks, Luigi Fabbri, morality, Communist party, Sam Dolgoff, united front, Ammon Hennacy, social ecology, commune (intentional community), Oscar Wilde, French Revolution, egoist anarchism, Comintern, transphobia, anarchism without adjectives, social control, means of production, Michel Onfray, Anarchism in France#The Fourth Republic (1945–1958), syndicalism, Anarchism in Spain, Iberian Anarchist Federation, International of Anarchist Federations, Emma Goldman, Netherlands, anarchist free school, International Workingmen's Association, Queer anarchism, Cantonal Revolution, trade unionism, Karl Marx, LGBT community, humanism, Anti-fascism, Carrara, political philosophy, Anarcho-transhumanism, libertarian socialist, Russian Revolution (1917), Two Cheers for Anarchism: Six Easy Pieces on Autonomy, Dignity, and Meaningful Work and Play, Emile Armand, insurrectionary anarchism, individual, Zhuang Zhou, Free Territory, White movement, Greenwich Village, Virginia Bolten, transcendentalist, public choice theory, wikt:brigand, Issues in anarchism#Participation in statist democracy, free love, Mutualism (economic theory), Anarchist St. Imier International, censorship, federalist, 6 February 1934 crisis, biennio rosso, anti-clerical, centralism, Anarchism: A Documentary History of Libertarian Ideas, minarchism, James C. Scott, First International, homosexuality, political theology, spontaneous order, Oranienburg concentration camp, anarcho-communism, negative liberty, post-modernism, Anarchism in Italy, Leopold Kohr, union of egoists, counterculture, Miguel Gimenez Igualada, philosophical anarchism, International Libertarian Solidarity, homosexual, Counterculture of the 1960s, Errico Malatesta, strikebreaker, Workers' Party of Marxist Unification, Clifford Harper, Reification (fallacy), patriarchy, anarchist law, Apostle (Christian), market (economics), Summerhill School, positive liberty, socialism, feminism, Direct action, Melchor Rodríguez García, William Godwin, Nazi concentration camps, Synthesist anarchism, Margaret Anderson, Han Ryner, Federation of Organized Trades and Labor Unions, technology, Workers Solidarity Movement, Edmund Burke, Encyclopædia Britannica, state (polity), Herbert Read, Park Güell, utilitarian, far right leagues, Limited government, self-ownership, Pejorative, homophobia, Industrial Workers of the World, The Dispossessed, Hague Congress (1872), Stalinism, Reciprocity (cultural anthropology), Fernand Pelloutier, individualist anarchism in France, The False Principle of our Education, individualist anarchism, Pierre Monatte, Soviet Union, counter-economics, Rudolf Rocker, Anarchism and capitalism, Parma, Black Rose Books, lesbian, Arditi del Popolo, Emile Armand (1872–1962), who propounded the virtues of free love in the Parisian anarchist milieu of the early 20th century, collectivism, Development criticism, John Henry Mackay, Benoît Broutchoux, Illegalism, Laozi, feminist, Christiaan Cornelissen, Syndicalist Workers' Federation, anarcho-syndicalism, Andalusia, Renzo Novatore, trade union, autonomist marxism, dictatorship of the proletariat, Mujeres Libres, Voltairine de Cleyre, Post-anarchism, participatory economics, Confederación Nacional del Trabajo, Syncretic politics, direct democracy, Jean-Jacques Rousseau, Green anarchism, Surrealism, labour unions, A. S. Neill, christian anarchist, Bonnot Gang, Anti-capitalism, Anarchism in Brazil, simple living, enlightened self-interest, Confédération générale du travail, class conflict, International Workers' Day, Hébertists, Gerrard Winstanley, Francoism, anarcho-pacifist, Andrej Grubacic, individualist anarchist and social anarchist thinkers., April Carter, private property, penal colonies, Libertarian socialism, Camillo Berneri, Christian anarchism, transhumanism, Lucifer, the Light-Bearer, Edna St. Vincent Millay, unschooling, Leo Tolstoy, M. E. Lazarus, Spanish Anarchists, Buddhist anarchism, ideology, William McKinley, anarcho-primitivism, Francesc Pi i Margall, :Category:Anarchism by country, International Workers Association, Anarcho-capitalism, Lois Waisbrooker, wikt:Solidarity, Baja California, social revolution, Unione Sindacale Italiana, Lev Chernyi, Alex Comfort, Sonnenburg, Leon Czolgosz, Volin, utopian, Argentine Libertarian Federation, Nudism, Left-wing market anarchism, insurrection, definitional concerns in anarchist theory, infinitive, affinity group, World Trade Organization Ministerial Conference of 1999 protest activity, class struggle, nonviolence, John Zerzan, poststructuralist, Noam Chomsky, Second Fitna, Julian Beck, Philadelphes, League of Peace and Freedom, Fédération Anarchiste, Kronstadt rebellion, Cold War, André Breton, Silvio Gesell, libertarian anarchism, voluntary association, anti-globalisation movement, birth control, L. Susan Brown, anarcho-naturism, personal property, Roundhead, Harold Barclay, The Joy of Sex, Council communism, Lucía Sánchez Saornil, tyrannicide, Neopaganism, lois scélérates, Johann Most, Anarchist Catalonia, Albert Camus, Protests of 1968, Alexander II of Russia, Spain's economy, Federazione Anarchica Italiana, Cuba, German Revolution of 1918–1919, stirner, Property is theft, Situationist International, law and economics

      ```

  • Add support for SMART notation for TfidfModel. Fix #1785 (@markroxor, #1791)

    • Natural extension of TfidfModel to allow different weighting and normalization schemes ```python from gensim.corpora import Dictionary from gensim.models import TfidfModel import gensim.downloader as api

      data = api.load("text8") dct = Dictionary(data) corpus = [dct.doc2bow(line) for line in data]

      Train Tfidf model using the SMART notation, smartirs="ntc" where

      'n' - natural term frequency

      't' - idf document frequency

      'c' - cosine normalization

      More information about possible values available in documentation or https://nlp.stanford.edu/IR-book/html/htmledition/document-and-query-weighting-schemes-1.html

      model = TfidfModel(corpus, id2word=dct, smartirs="ntc") vectorized_corpus = list(model[corpus])

      ```

    • SMART Information Retrieval System (wiki)

  • Add CircleCI for building Gensim documentation. Fix #1807 (@menshikh-iv, #1822)

    • An easy way to preview the rendered documentation (especially, if don't use Linux)
      • Go to "Details" link of CircleCI in your PR, click on the "Artifacts" tab, choose the HTML file that you want to view; a new tab will open with the rendered HTML page
    • Integration with Github, to see the documentation directly from the pull request page

:redcircle: Bug fixes: * Fix import in `getmyip. Fix #1771 ([@darindf](https://github.com/darindf), [#1772](https://github.com/RaRe-Technologies/gensim/pull/1772)) * Fix tox.ini/setup.cfg configuration ([@menshikh-iv](https://github.com/menshikh-iv), [#1815](https://github.com/RaRe-Technologies/gensim/pull/1815)) * Fix formula ingensim.summarization.bm25. Fix #1828 ([@sj29-innovate](https://github.com/sj29-innovate), [#1833](https://github.com/RaRe-Technologies/gensim/pull/1833)) * Fix the train method ofTranslationMatrix([@robotcator](https://github.com/robotcator), [#1838](https://github.com/RaRe-Technologies/gensim/pull/1838)) * Fix positional params used forgensim.models.CoherenceModelingensim.models.callbacks([@Alexjmsherman](https://github.com/Alexjmsherman), [#1823](https://github.com/RaRe-Technologies/gensim/pull/1823)) * Fix parameter setting forFastText.train. Fix #1818 ([@sj29-innovate](https://github.com/sj29-innovate), [#1837](https://github.com/RaRe-Technologies/gensim/pull/1837)) * Pin python2 explicitly for building documentation ([@menshikh-iv](https://github.com/menshikh-iv), [#1840](https://github.com/RaRe-Technologies/gensim/pull/1840)) * Remove dispatcher deadlock for distributed LDA ([@darindf](https://github.com/darindf), [#1817](https://github.com/RaRe-Technologies/gensim/pull/1817)) * FixscorefunctionfromLexicalEntailmentEvaluation`. Fix #1858 (@hachibaka, #1863) * Fix symmetrical case for hellinger distance. Fix #1854 (@caiyulun, #1860) * Remove wrong logging at import. Fix #1706 (@menshikh-iv, #1871)

:books: Tutorial and doc improvements: * Refactor documentation API Reference for gensim.summarization (@yurkai & @menshikh-iv, #1709) * Fix docstrings for gensim.similarities.index. Partial fix #1666 (@menshikh-iv, #1681) * Fix docstrings for gensim.models.translation_matrix (@KokuKUSIAKU & @menshikh-iv, #1806) * Fix docstrings for gensim.models.rpmodel (@jazzmuesli & @menshikh-iv, #1802) * Fix docstrings for gensim.utils (@kakshay21 & @menshikh-iv, #1797) * Fix docstrings for gensim.matutils (@Cheukting & @menshikh-iv, #1804) * Fix docstrings for gensim.models.logentropy_model (@minggli & @menshikh-iv, #1803) * Fix docstrings for gensim.models.normmodel (@AustenLamacraft & @menshikh-iv, #1805) * Refactor API reference gensim.topic_coherence. Fix #1669 (@CLearERR & @menshikh-iv, #1714) * Fix documentation for gensim.corpora.dictionary and gensim.corpora.hashdictionary. Partial fix #1671 (@CLearERR & @menshikh-iv, #1814) * Fix documentation for gensim.corpora. Partial fix #1671 (@anotherbugmaster & @menshikh-iv, #1729) * Update banner in doc pages (@piskvorky, #1865) * Fix errors in the doc2vec-lee notebook (@PeterHamilton, #1841) * Add wordnet mammal train file for Poincare notebook (@jayantj, #1781) * Update Poincare notebooks (#1774) (@jayantj, #1774) * Update contributing guide. Fix #1786 (@menshikh-iv, #1793) * Add model_to_dict one-liner to word2vec notebook. Fix #1269 (@kakshay21, #1776) * Add word embedding viz to word2vec notebook. Fix #1419 (@markroxor, #1800) * Fix description of sg parameter for gensim.models.FastText (@akutuzov, #1801) * Fix typo in doc2vec-IMDB. Fix #1788 (@apoorvaeternity, #1796) * Remove outdated bz2 examples from tutorials2 * Remove outdated bz2 + MmCorpus examples from tutorials (@menshikh-iv, #1867)

:+1: Improvements: * Refactor tests for gensim.corpora.WikiCorpus (@steremma, #1821)

:warning: Deprecations (will be removed in the next major release) * Remove - gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation) - gensim.examples - gensim.nosy - gensim.scripts.word2vec_standalone - gensim.scripts.make_wiki_lemma - gensim.scripts.make_wiki_online - gensim.scripts.make_wiki_online_lemma - gensim.scripts.make_wiki_online_nodebug - gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation) - "deprecated" functions and attributes

  • Move
    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

3.2.0, 2017-12-09

:star2: New features:

  • New download API for corpora and pre-trained models (@chaitaliSaini & @menshikh-iv, #1705 & #1632 & #1492)

    • Download large NLP datasets in one line of Python, then use with memory-efficient data streaming: ```python import gensim.downloader as api

      for article in api.load("wiki-english-20171001"): pass

      ```

    • Don’t waste time searching for good word embeddings, use the curated ones we included: ```python import gensim.downloader as api

      model = api.load("glove-twitter-25") model.most_similar("engineer")

      [('specialist', 0.957542896270752),

      ('developer', 0.9548177123069763),

      ('administrator', 0.9432312846183777),

      ('consultant', 0.93915855884552),

      ('technician', 0.9368376135826111),

      ('analyst', 0.9342101216316223),

      ('architect', 0.9257484674453735),

      ('engineering', 0.9159940481185913),

      ('systems', 0.9123805165290833),

      ('consulting', 0.9112802147865295)]

      ```

    • Blog post introducing the API and design decisions.

    • Notebook with examples

  • New model: PoincarĂŠ embeddings (@jayantj, #1696 & #1700 & #1757 & #1734)

    • Embed a graph (taxonomy) in the same way as word2vec embeds words: ```python from gensim.models.poincare import PoincareRelations, PoincareModel from gensim.test.utils import datapath

      data = PoincareRelations(datapath('poincarehypernyms.tsv')) model = PoincareModel(data) model.kv.mostsimilar("cat.n.01")

      [('kangaroo.n.01', 0.010581353439700418),

      ('gib.n.02', 0.011171531439892076),

      ('striped_skunk.n.01', 0.012025106076442395),

      ('metatherian.n.01', 0.01246679759214648),

      ('mammal.n.01', 0.013281303506525968),

      ('marsupial.n.01', 0.013941330203709653)]

      ```

    • Tutorial notebook on PoincarĂŠ embeddings

    • Model introduction and the journey of its implementation

    • Original paper on arXiv

  • Optimized FastText (@manneshiva, #1742)

    • New fast multithreaded implementation of FastText, natively in Python/Cython. Deprecates the existing wrapper for Facebook’s C++ implementation. ```python import gensim.downloader as api from gensim.models import FastText

    model = FastText(api.load("text8")) model.most_similar("cat")

    [('catnip', 0.8538144826889038),

    ('catwalk', 0.8136177062988281),

    ('catchy', 0.7828493118286133),

    ('caf', 0.7826495170593262),

    ('bobcat', 0.7745151519775391),

    ('tomcat', 0.7732658386230469),

    ('moat', 0.7728310823440552),

    ('caye', 0.7666271328926086),

    ('catv', 0.7651021480560303),

    ('caveat', 0.7643581628799438)]

    ```

  • Binary pre-compiled wheels for Windows, OSX and Linux (@menshikh-iv, MacPython/gensim-wheels/#7)

    • Users no longer need to have a C compiler for using the fast (Cythonized) version of word2vec, doc2vec, etc.
    • Faster Gensim pip installation

  • Added DeprecationWarnings to deprecated methods and parameters, with a clear schedule for removal.

:+1: Improvements: * Add Montemurro and Zanette's entropy based keyword extraction algorithm. Fix #665 (@PeteBleackley, #1738) * Fix flake8 E731, E402, refactor tests & sklearn API code. Partial fix #1644 (@horpto, #1689) * Reduce distribution size. Fix #1698 (@menshikh-iv, #1699) * Improve scan_vocab speed, build_vocab_from_freq method (@jodevak, #1695) * Improve segment_wiki script (@piskvorky, #1707) * Add custom dtype support for LdaModel. Partially fix #1576 (@xelez, #1656) * Add doc2idx method for gensim.corpora.Dictionary. Fix #1634 (@roopalgarg, #1720) * Add tox and pytest to gensim, integration with Travis and Appveyor. Fix #1613, #1644 (@menshikh-iv, #1721) * Add flag for hiding outdated data for gensim.downloader.info (@menshikh-iv, #1736) * Add reproducible order between python versions for gensim.corpora.Dictionary (@formi23, #1715) * Update tox.ini, setup.cfg, README.md (@menshikh-iv, #1741) * Add custom logsumexp for LdaModel (@arlenk, #1745)

:redcircle: Bug fixes: * Fix ranking formula in gensim.summarization.bm25. Fix #1718 (@souravsingh, #1726) * Fixed incompatibility in persistence for FastText wrapper. Fix #1642 (@chinmayapancholi13, #1723) * Fix `gensim.sklearnapibug withdocumentscolumnsparameter. Fix #1676 ([@chinmayapancholi13](https://github.com/chinmayapancholi13), [#1704](https://github.com/RaRe-Technologies/gensim/pull/1704)) * Fix slowdown of CI, remove pytest-cov ([@menshikh-iv](https://github.com/menshikh-iv), [#1728](https://github.com/RaRe-Technologies/gensim/pull/1728)) * Replace outdated packages in Dockerfile ([@rbahumi](https://github.com/rbahumi), [#1730](https://github.com/RaRe-Technologies/gensim/pull/1730)) * ReplacenumwordstotopninLdaMallet.showtopics. Fix #1747 ([@apoorvaeternity](https://github.com/apoorvaeternity), [#1749](https://github.com/RaRe-Technologies/gensim/pull/1749)) * Fixos.renamefromgensim.downloaderwhen 'src' and 'dst' on different partitions ([@anotherbugmaster](https://github.com/anotherbugmaster), [#1733](https://github.com/RaRe-Technologies/gensim/pull/1733)) * FixDeprecationWarningfromlogsumexp([@dreamgonfly](https://github.com/dreamgonfly), [#1703](https://github.com/RaRe-Technologies/gensim/pull/1703)) * Fix backward compatibility problem inPhrases.load. Fix #1751 ([@alexgarel](https://github.com/alexgarel), [#1758](https://github.com/RaRe-Technologies/gensim/pull/1758)) * Fixloadword2vecformatfromFastText. Fix #1743 ([@manneshiva](https://github.com/manneshiva), [#1755](https://github.com/RaRe-Technologies/gensim/pull/1755)) * Fix ipython kernel version inDockerfile. Fix #1762 ([@rbahumi](https://github.com/rbahumi), [#1764](https://github.com/RaRe-Technologies/gensim/pull/1764)) * Fix writing insegmentwiki([@horpto](https://github.com/horpto), [#1763](https://github.com/RaRe-Technologies/gensim/pull/1763)) * Fix write method of file requires byte-like object insegmentwiki([@horpto](https://github.com/horpto), [#1750](https://github.com/RaRe-Technologies/gensim/pull/1750)) * Fix incorrect vectors learned during online training forFastText. Fix #1752 ([@manneshiva](https://github.com/manneshiva), [#1756](https://github.com/RaRe-Technologies/gensim/pull/1756)) * Fixdtypeofmodel.wv.syn0vocabon updatingvocabforFastText. Fix #1759 ([@manneshiva](https://github.com/manneshiva), [#1760](https://github.com/RaRe-Technologies/gensim/pull/1760)) * Fix hashing-trick fromFastText.build_vocab. Fix #1765 ([@manneshiva](https://github.com/manneshiva), [#1768](https://github.com/RaRe-Technologies/gensim/pull/1768)) * Add explicitDeprecationWarningfor all outdated stuff. Fix #1753 ([@menshikh-iv](https://github.com/menshikh-iv), [#1769](https://github.com/RaRe-Technologies/gensim/pull/1769)) * Fix epsilon according todtypeinLdaModel` (@menshikh-iv, #1770)

:books: Tutorial and doc improvements: * Update perf numbers of segment_wiki (@piskvorky, #1708) * Update docstring for gensim.summarization.summarize. Fix #1575 (@fbarrios, #1702) * Refactor API Reference for gensim.parsing. Fix #1664 (@CLearERR, #1684) * Fix typos in doc2vec-wikipedia notebook (@youqad, #1727) * Fix PyPI long description rendering (@edigaryev, #1739) * Fix twitter badge src (@menshikh-iv) * Fix maillist badge color (@menshikh-iv)

:warning: Deprecations (will be removed in the next major release) * Remove - gensim.examples - gensim.nosy - gensim.scripts.word2vec_standalone - gensim.scripts.make_wiki_lemma - gensim.scripts.make_wiki_online - gensim.scripts.make_wiki_online_lemma - gensim.scripts.make_wiki_online_nodebug - gensim.scripts.make_wiki

  • Move
    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

3.1.0, 2017-11-06

:star2: New features: * Massive optimizations to LSI model training (@isamaru, #1620 & #1622) - LSI model allows use of single precision (float32), to consume 40% less memory while being 40% faster. - LSI model can now also accept CSC matrix as input, for further memory and speed boost. - Overall, if your entire corpus fits in RAM: 3x faster LSI training (SVD) in 4x less memory! ```python # just an example; the corpus stream is up to you streamingcorpus = gensim.corpora.MmCorpus("mytfidf_corpus.mm.gz")

# convert your corpus to a CSC sparse matrix (assumes the entire corpus fits in RAM)
in_memory_csc_matrix = gensim.matutils.corpus2csc(streaming_corpus, dtype=np.float32)

# then pass the CSC to LsiModel directly
model = LsiModel(corpus=in_memory_csc_matrix, num_topics=500, dtype=np.float32)
```
  • Even if you continue to use streaming corpora (your training dataset is too large for RAM), you should see significantly faster processing times and a lower memory footprint. In our experiments with a very large LSI model, we saw a drop from 29 GB peak RAM and 38 minutes (before) to 19 GB peak RAM and 26 minutes (now): python model = LsiModel(corpus=streaming_corpus, num_topics=500, dtype=np.float32)

  • Phrases allows to use common terms in bigrams. Before, if you are searching to reveal ngrams like car_with_driver and car_without_driver, you can either remove stop words before processing, but you will only find car_driver, or you won't find any of those forms (because they have three words, but also because high frequency of with will avoid them to be scored correctly), inspired by ES common grams token filter. ```python phrold = Phrases(corpus) phrnew = Phrases(corpus, common_terms=stopwords.words('en'))

    print(phrold[["we", "provide", "car", "with", "driver"]]) # ["we", "provide", "carwith", "driver"] print(phrnew[["we", "provide", "car", "with", "driver"]]) # ["we", "provide", "carwith_driver"] ```

  • CLI script for processing a raw Wikipedia dump (the xml.bz2 format provided by WikiMedia) to extract its articles in a plain text format. It extracts each article's title, section names and section content and saves them as json-line: bash python -m gensim.scripts.segment_wiki -f enwiki-latest-pages-articles.xml.bz2 | gzip > enwiki-latest-pages-articles.json.gz Processing the entire English Wikipedia dump (13.5 GB, link here) takes about 2.5 hours (i7-6700HQ, SSD).

    The output format is one article per line, serialized into JSON: python for line in smart_open('enwiki-latest-pages-articles.json.gz'): # read the file we just created article = json.loads(line) print("Article title: %s" % article['title']) for section_title, section_text in zip(article['section_titles'], article['section_texts']): print("Section title: %s" % section_title) print("Section text: %s" % section_text)

:+1: Improvements: * Speedup FastText tests (@horpto, #1686) * Add optimization for SlicedCorpus.len (@horpto, #1679) * Make word_vec return immutable vector. Fix #1651 (@CLearERR, #1662) * Drop Win x32 support & add rolling builds (@menshikh-iv, #1652) * Fix scoring function in Phrases. Fix #1533, #1635 (@michaelwsherman, #1573) * Add configuration for flake8 to setup.cfg (@mcobzarenco, #1636) * Add build_vocab_from_freq to Word2Vec, speedup scan_vocab (@jodevak, #1599) * Add most_similar_to_given method for KeyedVectors (@TheMathMajor, #1582) * Add getitem method to Sparse2Corpus to allow direct queries (@isamaru, #1621)

:redcircle: Bug fixes: * Add single core mode to CoherenceModel. Fix #1683 (@horpto, #1685) * Fix ResourceWarnings in tests. Partially fix #1519 (@horpto, #1660) * Fix DeprecationWarnings generated by deprecated assertEquals. Partial fix #1519 (@poornagurram, #1658) * Fix DeprecationWarnings for regex string literals. Fix #1646 (@franklsf95, #1649) * Fix pagerank algorithm. Fix #805 (@xelez, #1653) * Fix FastText inconsistent dtype. Fix #1637 (@mcobzarenco, #1638) * Fix `testfilename_filtering` test (@nehaljwani, #1647)

:books: Tutorial and doc improvements: * Fix code/docstring style (@menshikh-iv, #1650) * Update error message for supervised FastText. Fix #1498 (@ElSaico, #1645) * Add "DOI badge" to README. Fix #1610 (@dphov, #1639) * Remove duplicate annoy notebook. Fix #1415 (@Karamax, #1640) * Fix duplication and wrong markup in docs (@horpto, #1633) * Refactor dendrogram & topic network notebooks (@parulsethi, #1571) * Fix release badge (@menshikh-iv, #1631)

:warning: Deprecation part (will come into force in the next major release) * Remove - gensim.examples - gensim.nosy - gensim.scripts.word2vec_standalone - gensim.scripts.make_wiki_lemma - gensim.scripts.make_wiki_online - gensim.scripts.make_wiki_online_lemma - gensim.scripts.make_wiki_online_nodebug - gensim.scripts.make_wiki

  • Move
    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

Also, we'll create experimental subpackage for unstable models. Specific lists will be available in the next major release.

3.0.1, 2017-10-12

:redcircle: Bug fixes: * Fix Keras import, speedup importing time. Fix #1614 (@menshikh-v, #1615) * Fix Sphinx warnings and retreive all missing .rst (@anotherbugmaster and @menshikh-iv, #1612) * Fix logger message in lsidispatcher (@lorosanu, #1603)

:books: Tutorial and doc improvements: * Fix spelling (@jberkel, #1625)

:warning: Deprecation part (will come into force in the next release) * Remove - gensim.examples - gensim.nosy - gensim.scripts.word2vec_standalone - gensim.scripts.make_wiki_lemma - gensim.scripts.make_wiki_online - gensim.scripts.make_wiki_online_lemma - gensim.scripts.make_wiki_online_nodebug - gensim.scripts.make_wiki

  • Move
    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

Also, we'll create experimental subpackage for unstable models. Specific lists will be available in the next release.

3.0.0, 2017-09-27

:star2: New features: * Add unsupervised FastText to Gensim (@chinmayapancholi13, #1525) * Add sklearn API for gensim models (@chinmayapancholi13, #1462) * Add callback metrics for LdaModel and integration with Visdom (@parulsethi, #1399) * Add TranslationMatrix model (@robotcator, #1434) * Add word2vec-based coherence. Fix #1380 (@macks22, #1530)

:+1: Improvements: * Add 'diagonal' parameter for LdaModel.diff (@parulsethi, #1448) * Add 'score' function for SklLdaModel (@chinmayapancholi13, #1445) * Update sklearn API for gensim models (@chinmayapancholi13, #1473) [:warning: breaks backward compatibility] * Add CoherenceModel to LdaModel.top_topics. Fix #1128 (@macks22, #1427) * Add dendrogram viz for topics and JS metric (@parulsethi, #1484) * Add topic network viz (@parulsethi, #1536) * Replace viewitems to iteritems. Fix #1495 (@HodorTheCoder, #1508) * Fix Travis config and add style-checking for Ipython Notebooks. Fix #1518, #1520 (@menshikh-iv, #1522) * Remove mutable args from definitions. Fix #1561 (@zsef123, #1562) * Add Appveyour for all PRs. Fix #1565 (@menshikh-iv, #1565) * Refactor code by PEP8. Partially fix #1521 (@zsef123, #1550) * Refactor code by PEP8 with additional limitations. Fix #1521 (@menshikh-iv, #1569) * Update FastTextKeyedVectors.__contains__ (@ELind77, #1499) * Update WikiCorpus tokenization. Fix #1534 (@roopalgarg, #1537)

:redcircle: Bug fixes: * Remove round in LdaSeqModel.printtopic. Fix #1480 (@menshikh-iv, #1547) * Fix TextCorpus.samletext (@menshikh-iv, #1548) * Fix Mallet wrapper and tests for HDPTransform (@menshikh-iv, #1555) * Fix incorrect initialization ShardedCorpus with a generator. Fix #1511 (@karkkainenk1, #1512) * Add verification when summarizecorpus returns null. Fix #1531 (@fbarrios, #1570) * Fix doctag unicode problem. Fix 1543 (@englhardt, #1544) * Fix Translation Matrix (@robotcator, #1594) * Add trainable flag to KeyedVectors.getembeddinglayer. Fix #1557 (@zsef123, #1558)

:books: Tutorial and doc improvements: * Update exception text in TextCorpus.samletext. Partial fix #308 (@vlejd, #1444) * Remove extra filtertoken from tutorial (@VorontsovIE, #1502) * Update Doc2Vec-IMDB notebook (@pahdo, #1476) * Add Google Tag Manager for site (@yardos, #1556) * Update docstring explaining lack of multistream support in WikiCopus. Fix #1496 (@polm and @menshikh-iv, #1515) * Fix PathLineSentences docstring (@gojomo) * Fix typos from Translation Matrix notebook (@robotcator, #1598)

2.3.0, 2017-07-25

:star2: New features: * Add Dockerfile for gensim with external wrappers (@parulsethi, #1368) * Add sklearn wrapper for Word2Vec (@chinmayapancholi13, #1437) * Add loss function for Word2Vec. Fix #999 (@chinmayapancholi13, #1201) * Add sklearn wrapper for AuthorTopic model (@chinmayapancholi13, #1403)

:+1: Improvements: * Remove unittest2 (@souravsingh, #1490) * Add multiple scoring methods for Phrases. Partial fix #1363 (@michaelwsherman, #1464) * Add WordRank wrapper to Dockerfile (@parulsethi, #1460) * Add PathLineSentences. Fix #1364 (@michaelwsherman, #1423) * Add TextDirectoryCorpus and refactor TextCorpus. Fix #1387 (@macks22, #1459) * Add sparse input support with topn parameter in any2sparse. Fix #1294 (@manneshiva, #1321) * Add seed and length for sampletext. Partial fix #308 (@vlejd, #1422) * Add wordngram parameter to FastText (@fsonntag, #1432)

:redcircle: Bug fixes: * Fix fastText loading from .bin file. Fix #1236 (@prakhar2b, #1341) * Fix paths in WordRank and running gensim version in Dockerfile (@parulsethi, #1503) * Fix commit version for gensim in Dockerfile (@parulsethi, #1491) * Fix encoding problems with tests on windows. Fix #1441 (@menshikh-iv, #1469) * Fix parameters in scorecbowpair (@jmhessel, #1468) * Fix parameters in scoresentence_cbow (@jmhessel, #1467) * Fix TextDirectoryCorpus on windows (@macks22, #1463) * Fix gensim version in Dockerfile (@parulsethi, #1456) * Fix WordOccurenceAccumulator on windows. Fix #1441 (@macks22, #1449) * Fix scipy/numpy requirements (downgrade). Fix #1450 (@menshikh-iv, #1450)

:books: Tutorial and doc improvements: * Fix links and spaces in quick start guide (@iamsanten, #1500) * Fix error of ConcatedDoc2Vec in doc2vec-imdb notebook (@robocator, #1377) * Fix Sphinx warnings. Fix #1192 (@prerna135, #1442) * Fix typo in LdaModel.diff method (@parulsethi, #1461) * Add Tensorboard visualization for LDA (@parulsethi, #1396) * Update old and add new notebook with CoherenceModel (@macks22, #1431)

2.2.0, 2017-06-21

:star2: New features: * Add sklearn wrapper for RpModel (@chinmayapancholi13, #1395) * Add sklearn wrappers for LdaModel and LsiModel (@chinmayapancholi13, #1398) * Add sklearn wrapper for LdaSeq (@chinmayapancholi13, #1405) * Add keras wrapper for Word2Vec model (@chinmayapancholi13, #1248) * Add LdaModel.diff method (@menshikh-iv, #1334) * Allow use of truncated Dictionary for coherence measures. Fix #1342 (@macks22, #1349)

:+1: Improvements: * Fix saveastext/loadastext for Dictionary (@vlejd, #1402) * Add sampling support for corpus. Fix #308 (@vlejd, #1408) * Add napoleon extension to sphinx (@rasto2211, #1411) * Add KeyedVectors support to AnnoyIndexer (@quole, #1318) * Add BaseSklearnWrapper (@chinmayapancholi13, #1383) * Replace numwords to topn in model for unification. Fix #1198 (@prakhar2b, #1200) * Rename outpath to outname & add logging for WordRank model. Fix #1310 (@parulsethi, #1332) * Remove multiple iterations of corpus in pbooleandocument (@danielchamberlain, #1325) * Fix codestyle in TfIdf (@piskvorky, #1313) * Fix warnings from Sphinx. Partial fix #1192 (@souravsingh, #1330) * Add testenv to setup.py (@menshikh-iv, #1336)

:redcircle: Bug fixes: * Add cleanup in annoy test (@prakhar2b, #1420) * Add cleanup in lda backprop test (@prakhar2b, #1417) * Fix out-of-vocab in FastText (@jayantj, #1409) * Add cleanup in WordRank test (@parulsethi, #1410) * Fix rest requirements in Travis. Partial fix #1393 (@ibrahimsharaf, @menshikh-iv, #1400) * Fix morfessor exception. Partial fix #1324 (@souravsingh, #1406) * Fix test for FastText (@prakhar2b, #1371) * Fix WikiCorpus (@alekol, #1333) * Fix backward incompatibility for LdaModel (@chinmayapancholi13, #1327) * Fix support for old and new FastText model format. Fix #1301 (@prakhar2b, #1319) * Fix wrapper tests. Fix #1323 (@shubhamjain74, #1359) * Update exportphrases method. Fix #794 (@toumorokoshi, #1362) * Fix sklearn exception in test (@souravsingh, #1350)

:books: Tutorial and doc improvements: * Fix incorrect link in tutorials (@aneesh-joshi, #1426) * Add notebook with sklearn wrapper examples (@chinmayapancholi13, #1428) * Replace absolute pathes to relative in notebooks (@vochicong, #1414) * Fix code-style in keras notebook (@chinmayapancholi13, #1394) * Replace absolute pathes to relative in notebooks (@vochicong, #1407) * Fix typo in quickstart guide (@vochicong, #1404) * Update docstring for WordRank. Fix #1384 (@parulsethi, #1378) * Update docstring for SkLdaModel (@chinmayapancholi13, #1382) * Update logic for updatetype in LdaModel (@chinmayapancholi13, #1389) * Update docstring for Doc2Vec (@jstol, #1379) * Fix docstring for KL-distance (@viciousstar, #1373) * Update CorporaandVectorSpaces tutorial (@charliejharrison, #1308) * Add visualization for difference between LdaModel (@menshikh-iv, #1374) * Fix punctuation & typo in changelog (@piskvorky, @menshikh-iv, #1366) * Fix PEP8 & typo in several PRs (@menshikh-iv, #1369) * Update docstrings connected with backward compability in for LdaModel (@chinmayapancholi13, #1365) * Update CorporaandVectorSpaces tutorial (@schuyler1d, #1360) * Fix typo in Doc2Vec doctsring (@fujiyuu75, #1356) * Update Annoy tutorial (@pmbaumgartner, #1355) * Update temp folder in tutorials (@yl2526, #1352) * Remove spaces after print in TopicsandTransformation tutorial (@gsimore, #1354) * Update Dictionary docstring (@oonska, #1347) * Add section headings in word2vec notebook (@MikeTheReader, #1348) * Fix broken urls in starter tutorials (@ka7eh, #1346) * Update quick start notebook (@yardsale8, #1345) * Fix typo in quick start notebook (@MikeTheReader, #1344) * Fix docstring in keyedvectors (@chinmayapancholi13, #1337)

2.1.0, 2017-05-12

:star2: New features: * Add modified saveword2vecformat for Doc2Vec, to save document vectors. (@parulsethi, #1256)

:+1: Improvements: * Add automatic code style check limited only to the code modified in PR (@tmylk, #1287) * Replace logger.warn by logger.warning (@chinmayapancholi13, #1295) * Docs word2vec docstring improvement, deprecation labels (@shubhvachher, #1274) * Stop passing 'sentences' as parameter to Doc2Vec. Fix #511 (@gogokaradjov, #1306)

:red_circle: Bug fixes: * Allow indexing with np.int64 in doc2vec. Fix #1231 (@bogdanteleaga, #1254) * Update Doc2Vec docstring. Fix #1302 (@datapythonista, #1307) * Ignore rst and ipynb file in Travis flake8 validations (@datapythonista, #1309)

:books: Tutorial and doc improvements: * Update Tensorboard Doc2Vec notebook (@parulsethi, #1286) * Update Doc2Vec IMDB Notebook, replace codesc to smart_open (@robotcator, #1278) * Add explanation of size to Word2Vec Notebook (@jbcoe, #1305) * Add extra param to WordRank notebook. Fix #1276 (@parulsethi, #1300) * Update warning message in WordRank (@parulsethi, #1299)

2.0.0, 2017-04-10

Breaking changes:

Any direct calls to method train() of Word2Vec/Doc2Vec now require an explicit epochs parameter and explicit estimate of corpus size. The most usual way to call train is vec_model.train(sentences, total_examples=self.corpus_count, epochs=self.iter) See the method documentation for more information.

  • Explicit epochs and corpus size in word2vec train(). (@gojomo, @robotcator, #1139, #1237)

New features: * Add output word prediction in word2vec. Only for negative sampling scheme. See ipynb (@chinmayapancholi13, #1209) * scikitlearn wrapper for LSI Model in Gensim (@chinmayapancholi13, #1244) * Add the 'keeptokens' parameter to 'filter_extremes'. (@toliwa, #1210) * Load FastText models with specified encoding (@jayantj, #1210)

Improvements: * Fix loading large FastText models on Mac. (@jaksmid, #1196) * Sklearn LDA wrapper now works in sklearn pipeline (@kris-singh, #1213) * glove2word2vec conversion script refactoring (@parulsethi, #1247) * Word2vec error message when update called before train . Fix #1162 (@hemavakade, #1205) * Allow training if model is not modified by "minimizemodel". Add deprecation warning. (@chinmayapancholi13, #1207) * Update the warning text when building vocab on a trained w2v model (@prakhar2b, #1190)

Bug fixes:

  • Fix word2vec reset_from bug in v1.0.1 Fix #1230. (@Kreiswolke, #1234)

  • Distributed LDA: checking the length of docs instead of the boolean value, plus int index conversion (@saparina, #1191)

  • syn0lockf initialised with zero in intersectword2vec_format() (@KiddoZhu, #1267)

  • Fix wordrank maxiterdump calculation. Fix #1216 (@ajkl, #1217)

  • Make SgNegative test use skip-gram (@shubhvachher, #1252)

  • pep8/pycodestyle fixes for hanging indents in Summarization module (@SamriddhiJain, #1202)

  • WordRank and Mallet wrappers single vs double quote issue in windows. (@prakhar2b, #1208)

  • Fix #824 : no corpus in init, but trim_rule in init (@prakhar2b, #1186)

  • Hardcode version number. Fix #1138. (@tmylk, #1138)

Tutorial and doc improvements:

  • Color dictionary according to topic notebook update (@bhargavvader, #1164)

  • Fix hdp show_topic/s docstring (@parulsethi, #1264)

  • Add docstrings for word2vec.py forwarding functions (@shubhvachher, #1251)

  • updated description for worker_loop function used in score function (@chinmayapancholi13, #1206)

1.0.1, 2017-03-03

  • Rebuild cumulative table on load. Fix #1180. (@tmylk, #1181)
  • mostsimilarcosmul bug fix (@dkim010, #1177)
  • Fix loading old word2vec models pre-1.0.0 (@jayantj, #1179)
  • Load utf-8 words in fasttext (@jayantj, #1176)

1.0.0, 2017-02-24

New features: * Add Author-topic modeling (@olavurmortensen, #893) * Add FastText word embedding wrapper (@Jayantj, #847) * Add WordRank word embedding wrapper (@parulsethi, #1066, #1125) * Add VarEmbed word embedding wrapper (@anmol01gulati, #1067)) * Add sklearn wrapper for LDAModel (@AadityaJ, #932)

Deprecated features:

  • Move load_word2vec_format and save_word2vec_format out of Word2Vec class to KeyedVectors (@tmylk, #1107)
  • Move properties syn0norm, syn0, vocab, index2word from Word2Vec class to KeyedVectors (@tmylk,#1147)
  • Remove support for Python 2.6, 3.3 and 3.4 (@tmylk,#1145)

Improvements:

  • Python 3.6 support (@tmylk #1077)
  • Phrases and Phraser allow a generator corpus (ELind77 #1099)
  • Ignore DocvecsArray.doctag_syn0norm in save. Fix #789 (@accraze, #1053)
  • Fix bug in LsiModel that occurs when id2word is a Python 3 dictionary. (@cvangysel, #1103
  • Fix broken link to paper in readme (@bhargavvader, #1101)
  • Lazy formatting in evaluatewordpairs (@akutuzov, #1084)
  • Deacc option to keywords pre-processing (@bhargavvader, #1076)
  • Generate Deprecated exception when using Word2Vec.loadword2vecformat (@tmylk, #1165)
  • Fix hdpmodel constructor docstring for print_topics (#1152) (@toliwa, #1152)
  • Default to perwordtopics=False in LDA get_item for performance (@menshikh-iv, #1154)
  • Fix bound computation in Author Topic models. (@olavurmortensen, #1156)
  • Write UTF-8 byte strings in tensorboard conversion (@tmylk, #1144)
  • Make top_topics and sparse2full compatible with numpy 1.12 strictly int idexing (@tmylk, #1146)

Tutorial and doc improvements:

  • Clarifying comment in is_corpus func in utils.py (@greninja, #1109)
  • Tutorial TopicsandTransformations fix markdown and add references (@lgmoneda, #1120)
  • Fix doc2vec-lee.ipynb results to match previous behavior (@bahbbc, #1119)
  • Remove Pattern lib dependency in News Classification tutorial (@luizcavalcanti, #1118)
  • CorporaandVector_Spaces tutorial text clarification (@lgmoneda, #1116)
  • Update Transformation and Topics link from quick start notebook (@mariana393, #1115)
  • Quick Start Text clarification and typo correction (@luizcavalcanti, #1114)
  • Fix typos in Author-topic tutorial (@Fil, #1102)
  • Address benchmark inconsistencies in Annoy tutorial (@droudy, #1113)
  • Add note about Annoy speed depending on numpy BLAS setup in annoytutorial.ipynb (@greninja, #1137)
  • Fix dependencies description on doc2vec-IMDB notebook (@luizcavalcanti, #1132)
  • Add documentation for WikiCorpus metadata. (@kirit93, #1163)

1.0.0RC2, 2017-02-16

  • Add note about Annoy speed depending on numpy BLAS setup in annoytutorial.ipynb (@greninja, #1137)
  • Remove direct access to properties moved to KeyedVectors (@tmylk, #1147)
  • Remove support for Python 2.6, 3.3 and 3.4 (@tmylk, #1145)
  • Write UTF-8 byte strings in tensorboard conversion (@tmylk, #1144)
  • Make top_topics and sparse2full compatible with numpy 1.12 strictly int idexing (@tmylk, #1146)

1.0.0RC1, 2017-01-31

New features: * Add Author-topic modeling (@olavurmortensen, #893) * Add FastText word embedding wrapper (@Jayantj, #847) * Add WordRank word embedding wrapper (@parulsethi, #1066, #1125) * Add sklearn wrapper for LDAModel (@AadityaJ, #932)

Improvements: * Python 3.6 support (@tmylk #1077) * Phrases and Phraser allow a generator corpus (ELind77 #1099) * Ignore DocvecsArray.doctagsyn0norm in save. Fix #789 (@accraze, #1053) * Move load and save word2vecformat out of word2vec class to KeyedVectors (@tmylk, #1107) * Fix bug in LsiModel that occurs when id2word is a Python 3 dictionary. (@cvangysel, #1103 * Fix broken link to paper in readme (@bhargavvader, #1101) * Lazy formatting in evaluatewordpairs (@akutuzov, #1084) * Deacc option to keywords pre-processing (@bhargavvader, #1076)

Tutorial and doc improvements:

  • Clarifying comment in is_corpus func in utils.py (@greninja, #1109)
  • Tutorial TopicsandTransformations fix markdown and add references (@lgmoneda, #1120)
  • Fix doc2vec-lee.ipynb results to match previous behavior (@bahbbc, #1119)
  • Remove Pattern lib dependency in News Classification tutorial (@luizcavalcanti, #1118)
  • CorporaandVector_Spaces tutorial text clarification (@lgmoneda, #1116)
  • Update Transformation and Topics link from quick start notebook (@mariana393, #1115)
  • Quick Start Text clarification and typo correction (@luizcavalcanti, #1114)
  • Fix typos in Author-topic tutorial (@Fil, #1102)
  • Address benchmark inconsistencies in Annoy tutorial (@droudy, #1113)

0.13.4.1, 2017-01-04

  • Disable direct access warnings on save and load of Word2vec/Doc2vec (@tmylk, #1072)
  • Making Default hs error explicit (@accraze, #1054)
  • Removed unnecessary numpy imports (@bhargavvader, #1065)
  • Utils and Matutils changes (@bhargavvader, #1062)
  • Tests for the evaluatewordpairs function (@akutuzov, #1061)

0.13.4, 2016-12-22

  • Added suggested lda model method and print methods to HDP class (@bhargavvader, #1055)
  • New class KeyedVectors to store embedding separate from training code (@anmol01gulati and @droudy, #980)
  • Evaluation of word2vec models against semantic similarity datasets like SimLex-999 (@akutuzov, #1047)
  • TensorBoard word embedding visualisation of Gensim Word2vec format (@loretoparisi, #1051)
  • Throw exception if load() is called on instance rather than the class in word2vec and doc2vec (@dust0x, #889)
  • Loading and Saving LDA Models across Python 2 and 3. Fix #853 (@anmolgulati, #913, #1093)
  • Fix automatic learning of eta (prior over words) in LDA (@olavurmortensen, #1024).
    • eta should have dimensionality V (size of vocab) not K (number of topics). eta with shape K x V is still allowed, as the user may want to impose specific prior information to each topic.
    • eta is no longer allowed the "asymmetric" option. Asymmetric priors over words in general are fine (learned or user defined).
    • As a result, the eta update (update_eta) was simplified some. It also no longer logs eta when updated, because it is too large for that.
    • Unit tests were updated accordingly. The unit tests expect a different shape than before; some unit tests were redundant after the change; eta='asymmetric' now should raise an error.
  • Optimise showtopics to only call getlambda once. Fix #1006. (@bhargavvader, #1028)
  • HdpModel doc improvement. Inference and print_topics (@dsquareindia, #1029)
  • Removing Doc2Vec defaults so that it won't override Word2Vec defaults. Fix #795. (@markroxor, #929)
  • Remove warning on gensim import "pattern not installed". Fix #1009 (@shashankg7, #1018)
  • Add deletetemporarytraining_data() function to word2vec and doc2vec models. (@deepmipt-VladZhukov, #987)
  • Documentation improvements (@IrinaGoloshchapova, #1010, #1011)
  • LDA tutorial by Olavur, tips and tricks (@olavurmortensen, #779)
  • Add double quote in commmand line to run on Windows (@akarazeev, #1005)
  • Fix directory names in notebooks to be OS-independent (@mamamot, #1004)
  • Respect clipstart, clipend in most_similar. Fix #601. (@parulsethi, #994)
  • Replace Python sigmoid function with scipy in word2vec & doc2vec (@markroxor, #989)
  • WMD to return 0 instead of inf for sentences that contain a single word (@rbahumi, #986)
  • Pass all the params through the apply call in lda.getdocumenttopics(), test case to use the perwordtopics through the corpus in test_ldamodel (@parthoiiitm, #978)
  • Pyro annotations for lsi_worker (@markroxor, #968)

0.13.3, 2016-10-20

  • Add vocabulary expansion feature to word2vec. (@isohyt, #900)
  • Tutorial: Reproducing Doc2vec paper result on wikipedia. (@isohyt, #654)
  • Add Save/Load interface to AnnoyIndexer for index persistence (@fortiema, #845)
  • Fixed issue #938,Creating a unified base class for all topic models. (@markroxor, #946)
    • breaking change in HdpTopicFormatter.show_topics
  • Add Phraser for Phrases optimization. ( @gojomo & @anujkhare , #837)
  • Fix issue #743, in word2vec's n_similarity method if at least one empty list is passed ZeroDivisionError is raised (@pranay360, #883)
  • Change export_phrases in Phrases model. Fix issue #794 (@AadityaJ, #879)
    • bigram construction can now support multiple bigrams within one sentence
  • Fix issue #838, RuntimeWarning: overflow encountered in exp (@markroxor, #895)
  • Change some log messages to warnings as suggested in issue #828. (@rhnvrm, #884)
  • Fix issue #851, In summarizer.py, RunTimeError is raised if single sentence input is provided to avoid ZeroDivionError. (@metalaman, #887)
  • Fix issue #791, correct logic for iterating over SimilarityABC interface. (@MridulS, #839)
  • Fix RP model loading for large Fortran-order arrays (@piskvorky, #605)
  • Remove ShardedCorpus from init because of Theano dependency (@tmylk, #919)
  • Documentation improvements ( @dsquareindia & @tmylk, #914, #906 )
  • Add Annoy memory-mapping example (@harshul1610, #899)
  • Fixed issue #601, correct docID in most_similar for clip range (@parulsethi, #994)

0.13.2, 2016-08-19

  • wordtopics has changed to word_topics in ldamallet, and fixed issue #764. (@bhargavvader, #771)
    • assigning wordtopics value of word_topics to keep backward compatibility, for now
  • topics, topn parameters changed to numtopics and numwords in showtopics() and printtopics() (@droudy, #755)
    • In hdpmodel and dtmmodel
    • NOT BACKWARDS COMPATIBLE!
  • Added randomstate parameter to LdaState initializer and checkrandom_state() (@droudy, #113)
  • Topic coherence update with c_uci, c_npmi measures. LdaMallet, LdaVowpalWabbit support. Add topics parameter to coherencemodel. Can now provide tokenized topics to calculate coherence value. Faster backtracking. (@dsquareindia, #750, #793)
  • Added a check for empty (no words) documents before starting to run the DTM wrapper if model = "fixed" is used (DIM model) as this causes the an error when such documents are reached in training. (@eickho, #806)
  • New parameters limit, datatype for loadword2vecformat(); lockf for intersectword2vecformat (@gojomo, #817)
  • Changed use_lowercase option in word2vec accuracy to case_insensitive to account for case variations in training vocabulary (@jayantj, #804
  • Link to Doc2Vec on airline tweets example in tutorials page (@544895340, #823)
  • Small error on Doc2vec notebook tutorial (@charlessutton, #816)
  • Bugfix: Full2sparse clipped to use abs value (@tmylk, #811)
  • WMD docstring: add tutorial link and query example (@tmylk, #813)
  • Annoy integration to speed word2vec and doc2vec similarity. Tutorial update (@droudy, #799,#792 )
  • Add converter of LDA model between Mallet, Vowpal Wabit and gensim (@dsquareindia, #798, #766)
  • Distributed LDA in different network segments without broadcast (@menshikh-iv, #782)
  • Update CorporaandVector_Spaces.ipynb (@megansquire, #772)
  • DTM wrapper bug fixes caused by renaming num_words in #755 (@bhargavvader, #770)
  • Add LsiModel.docs_processed attribute (@hobson, #763)
  • Dynamic Topic Modelling in Python. Google Summer of Code 2016 project. (@bhargavvader, #739, #831)

0.13.1, 2016-06-22

  • Topic coherence Cv and Umass (@dsquareindia, #710)

0.13.0, 2016-06-21

  • Added Distance Metrics to matutils.pt (@bhargavvader, #656)
  • Tutorials migrated from website to ipynb (@j9chan, #721), (@jesford, #733), (@jesford, #725), (@jesford, #716)
  • New doc2vec intro tutorial (@seanlaw, #730)
  • Gensim Quick Start Tutorial (@andrewjlm, #727)
  • Add export_phrases(sentences) to model Phrases (hanabi1224 #588)
  • SparseMatrixSimilarity returns a sparse matrix if maintain_sparsity is True (@davechallis, #590)
  • added functionality for Topics of Words in document - i.e, dynamic topics. (@bhargavvader, #704)
    • also included tutorial which explains new functionalities, and document word-topic colring.
  • Made normalization an explicit transformation. Added 'l1' norm support (@dsquareindia, #649)
  • added term-topics API for most probable topic for word in vocab. (@bhargavvader, #706)
  • buildvocab takes progressper parameter for smaller output (@zer0n, #624)
  • Control whether to use lowercase for computing word2vec accuracy. (@alantian, #607)
  • Easy import of GloVe vectors using Gensim (Manas Ranjan Kar, #625)
    • Allow easy port of GloVe vectors into Gensim
    • Standalone script with command line arguments, compatible with Python>=2.6
    • Usage: python -m gensim.scripts.glove2word2vec -i glovevectors.txt -o outputword2vec_compatible.txt
  • Add similar_by_word() and similar_by_vector() to word2vec (@isohyt, #381)
  • Convenience method for similarity of two out of training sentences to doc2vec (@ellolo, #707)
  • Dynamic Topic Modelling Tutorial updated with Dynamic Influence Model (@bhargavvader, #689)
  • Added function to filter 'n' most frequent words from the dictionary (@abhinavchawla, #718)
  • Raise warnings if vocab is single character elements and if alpha is increased in word2vec/doc2vec (@dsquareindia, #705)
  • Tests for wikidump (@jonmcoe, #723)
  • Mallet wrapper sparse format support (@RishabGoel, #664)
  • Doc2vec pre-processing script translated from bash to Python (@andrewjlm, #720)

0.12.4, 2016-01-29

  • Better internal handling of job batching in word2vec (#535)
    • up to 300% speed up when training on very short documents (~tweets)
  • Word2vec CLI in line with original word2vec.c (Andrey Kutuzov, #538)
    • Same default values. See diff https://github.com/akutuzov/gensim/commit/6456cbcd75e6f8720451766ba31cc046b4463ae2
    • Standalone script with command line arguments matching those of original C tool.
    • Usage: python -m gensim.scripts.word2vecstandalone -train data.txt -output trainedvec.txt -size 200 -window 2 -sample 1e-4
  • Improved loadword2vecformat() performance (@svenkreiss, #555)
    • Remove init_sims() call for performance improvements when normalized vectors are not needed.
    • Remove norm_only parameter (API change). Call init_sims(replace=True) after the load_word2vec_format() call for the old norm_only=True behavior.
  • Word2vec allows non-strict unicode error handling (ignore or replace) (Gordon Mohr, #466)
  • Doc2Vec model.docvecs[key] now raises KeyError for unknown keys (Gordon Mohr, #520)
  • Fix DocvecsArray.index_to_doctag so most_similar() returns string doctags (Gordon Mohr, #560)
  • On-demand loading of the pattern library in utils.lemmatize (Jan Zikes, #461)
    • utils.HAS_PATTERN flag moved to utils.has_pattern()
  • Threadsafe Word2Vec/Doc2Vec finish-check to avoid hang/unending Word2Vec/Doc2Vec training (Gordon Mohr, #571)
  • Tuned TestWord2VecModel.test_cbow_hs() against random failures (Gordon Mohr, #531)
  • Prevent ZeroDivisionError when default_timer() indicate no elapsed time (Gordon Mohr, #518)
  • Forwards compatibility for NumPy > 1.10 (Matti Lyra, #494, #513)
    • LdaModel and LdaMulticore produce a large number of DeprecationWarnings from .inference() because the term ids in each chunk returned from utils.grouper are floats. This behaviour has been changed so that the term IDs are now ints.
    • utils.grouper returns a python list instead of a numpy array in .update() when LdaModel is called in non distributed mode
    • in distributed mode .update() will still call utils.grouper with as_numpy=True to save memory
    • LdaModel.update and LdaMulticore.update have a new keyword parameter chunksasnumpy=True/False (defaults to False) that allows controlling this behaviour

0.12.3, 2015-11-05

  • Make show_topics return value consistent across models (Christopher Corley, #448)
    • All models with the show_topics method should return a list of (topic_number, topic) tuples, where topic is a list of (word, probability) tuples.
    • This is a breaking change that affects users of the LsiModel, LdaModel, and LdaMulticore that may be reliant on the old tuple layout of (probability, word).
  • Mixed integer & string document-tags (keys to doc-vectors) will work (Gordon Mohr, #491)
    • DocvecsArray's index2doctag list is renamed/reinterpreted as offset2doctag
    • offset2doctag entries map to doctag_syn0 indexes after last plain-int doctag (if any)
    • (If using only string doctags, offset2doctag may be interpreted same as index2doctag.)
  • New Tutorials on Dynamic Topic Modelling and Classification via Word2Vec (@arttii #471, @mataddy #500)
  • Auto-learning for the eta parameter on the LdaModel (Christopher Corley, #479)
  • Python 3.5 support
  • Speed improvements to keyword and summarisation methods (@erbas #441)
  • OSX wheels (#504)
  • Win build (#492)

0.12.2, 2015-09-19

  • tutorial on text summarization (Ólavur Mortensen, #436)
  • more flexible vocabulary construction in word2vec & doc2vec (Philipp Dowling, #434)
  • added support for sliced TransformedCorpus objects, so that after applying (for instance) TfidfModel the returned corpus remains randomly indexable. (Matti Lyra, #425)
  • changed the LdaModel.save so that a custom ignore list can be passed in (Matti Lyra, #331)
  • added support for NumPy style fancy indexing to corpus objects (Matti Lyra, #414)
  • py3k fix in distributed LSI (spacecowboy, #433)
  • Windows fix for setup.py (#428)
  • fix compatibility for scipy 0.16.0 (#415)

0.12.1, 2015-07-20

  • improvements to testing, switch to Travis CI containers
  • support for loading old word2vec models (<=0.11.1) in 0.12+ (Gordon Mohr, #405)
  • various bug fixes to word2vec, doc2vec (Gordon Mohr, #393, #386, #404)
  • TextSummatization support for very short texts (Federico Barrios, #390)
  • support for word2vec[['word1', 'word2'...]] convenience API calls (Satish Palaniappan, #395)
  • MatrixSimilarity supports indexing generator corpora (single pass)

0.12.0, 2015-07-06

  • complete API, performance, memory overhaul of doc2vec (Gordon Mohr, #356, #373, #380, #384)
    • fast infer_vector(); optional memory-mapped doc vectors; memory savings with int doc IDs
    • 'dbowwords' for combined DBOW & word skip-gram training; new 'dmconcat' mode
    • multithreading & negative-sampling optimizations (also benefitting word2vec)
    • API NOTE: doc vectors must now be accessed/compared through model's 'docvecs' field (eg: "model.docvecs['myID']" or "model.docvecs.mostsimilar('my_ID')")
    • https://github.com/piskvorky/gensim/blob/develop/docs/notebooks/doc2vec-IMDB.ipynb
  • new "text summarization" module (PR #324: Federico Lopez, Federico Barrios)
    • https://github.com/summanlp/docs/raw/master/articulo/articulo-en.pdf
  • new matutils.argsort with partial sort
    • performance speedups to all similarity queries (word2vec, Similarity classes...)
  • word2vec can compute likelihood scores for classification (Mat Addy, #358)
    • http://arxiv.org/abs/1504.07295
    • http://nbviewer.ipython.org/github/taddylab/deepir/blob/master/w2v-inversion.ipynb
  • word2vec supports "encoding" parameter when loading from C format, for non-utf8 models
  • more memory-efficient word2vec training (#385)
  • fixes to Python3 compatibility (Pavel Kalaidin #330, S-Eugene #369)
  • enhancements to save/load format (Liang Bo Wang #363, Gordon Mohr #356)
    • pickle defaults to protocol=2 for better py3 compatibility
  • fixes and improvements to wiki parsing (Lukas Elmer #357, Excellent5 #333)
  • fix to phrases scoring (Ikuya Yamada, #353)
  • speed up of phrases generation (Dave Challis, #349)
  • changes to multipass LDA training (Christopher Corley, #298)
  • various doc improvements and fixes (Matti Lyra #331, Hongjoo Lee #334)
  • fixes and improvements to LDA (Christopher Corley #323)

0.11.0 = 0.11.1 = 0.11.1-1, 2015-04-10

  • added "topic ranking" to sort topics by coherence in LdaModel (jtmcmc, #311)
  • new fast ShardedCorpus out-of-core corpus (Jan Hajic jr., #284)
  • utils.smartopen now uses the smartopen package (#316)
  • new wrapper for LDA in Vowpal Wabbit (Dave Challis, #304)
  • improvements to the DtmModel wrapper (Yang Han, #272, #277)
  • move wrappers for external modeling programs into a submodule (Christopher Corley, #295)
  • allow transparent compression of NumPy files in save/load (Christopher Corley, #248)
  • save/load methods now accept file handles, in addition to file names (macks22, #292)
  • fixes to LdaMulticore on Windows (Feng Mai, #305)
  • lots of small fixes & py3k compatibility improvements (Chyi-Kwei Yau, Daniel Nouri, Timothy Emerick, Juarez Bochi, Christopher Corley, Chirag Nagpal, Jan Hajic jr., FlĂĄvio Codeço Coelho)
  • re-released as 0.11.1 and 0.11.1-1 because of a packaging bug

0.10.3, 2014-11-17

  • added streamed phrases = collocation detection (Miguel Cabrera, #258)
  • added param for multiple word2vec epochs (sebastienj, #243)
  • added doc2vec (=paragraph2vec = extension of word2vec) model (Timothy Emerick, #231)
  • initialize word2vec deterministically, for increased experiment reproducibility (KCzar, #240)
  • all indexed corpora now allow full Python slicing syntax (Christopher Corley, #246)
  • update distributed code for new Pyro4 API and py3k (Michael Brooks, Marco Bonzanini, #255, #249)
  • fixes to six module version (Lars Buitinck, #259)
  • fixes to setup.py (Maxim Avanov and Christopher Corley, #260, #251)
  • ...and lots of minor fixes & updates all around

0.10.2, 2014-09-18

  • new parallelized, LdaMulticore implementation (Jan Zikes, #232)
  • Dynamic Topic Models (DTM) wrapper (Arttii, #205)
  • word2vec compiled from bundled C file at install time: no more pyximport (#233)
  • standardize show/printtopics in LdaMallet (Benjamin Bray, #223)
  • add new word2vec multiplicative objective (3CosMul) of Levy & Goldberg (Gordon Mohr, #224)
  • preserve case in MALLET wrapper (mcburton, #222)
  • support for matrix-valued topic/word prior eta in LdaModel (mjwillson, #208)
  • py3k fix to SparseCorpus (Andreas Madsen, #234)
  • fix to LowCorpus when switching dictionaries (Christopher Corley, #237)

0.10.1, 2014-07-22

  • word2vec: new n_similarity method for comparing two sets of words (François Scharffe, #219)
  • make LDA print/show topics parameters consistent with LSI (Bram Vandekerckhove, #201)
  • add option for efficient word2vec subsampling (Gordon Mohr, #206)
  • fix length calculation for corpora on empty files (Christopher Corley, #209)
  • improve file cleanup of unit tests (Christopher Corley)
  • more unit tests
  • unicode now stored everywhere in gensim internally; accepted input stays either utf8 or unicode
  • various fixes to the py3k ported code
  • allow any dict-like input in Dictionary.from_corpus (Andreas Madsen)
  • error checking improvements to the MALLET wrapper
  • ignore non-articles during wiki parsig
  • utils.lemmatize now (optionally) ignores stopwords

0.10.0 (aka "PY3K port"), 2014-06-04

  • full Python 3 support (targeting 3.3+, #196)
  • all internal methods now expect & store unicode, instead of utf8
  • new optimized word2vec functionality: negative sampling, cbow (sebastien-j, #162)
  • allow by-frequency sort in Dictionary.saveastext (Renaud Richardet, #192)
  • add topic printing to HDP model (Tiepes, #190)
  • new gensim_addons package = optional install-time Cython compilations (BjĂśrn Esser, #197)
  • added py3.3 and 3.4 to Travis CI tests
  • fix a cbow word2vec bug (Liang-Chi Hsieh)

0.9.1, 2014-04-12

  • MmCorpus fix for Windows
  • LdaMallet support for printing/showing topics
  • fix LdaMallet bug when user specified a file prefix (Victor, #184)
  • fix LdaMallet output when input is single vector (Suvir)
  • added LdaMallet unit tests
  • more py3k fixes (Lars Buitinck)
  • change order of LDA topic printing (Fayimora Femi-Balogun, #188)

0.9.0, 2014-03-16

  • save/load automatically single out large arrays + allow mmap
  • allow .gz/.bz2 corpus filenames => transparently (de)compressed I/O
  • CBOW model for word2vec (SĂŠbastien Jean, #176)
  • new API for storing corpus metadata (Joseph Chang, #169)
  • new LdaMallet class = train LDA using wrapped Mallet
  • new MalletCorpus class for corpora in Mallet format (Christopher Corley, #179)
  • better Wikipedia article parsing (Joseph Chang, #170)
  • word2vec loadword2vecformat uses less memory (Yves Raimond, #164)
  • load/store vocabulary files for word2vec C format (Yves Raimond, #172)
  • HDP estimation on new documents (Elliot Kulakow, #153)
  • store labels in SvmLight corpus (Ritesh, #152)
  • fix word2vec binary load on Windows (Stephanus van Schalkwyk)
  • replace numpy.svd with scipy.svd for more stability (Sven DĂśring, #159)
  • parametrize LDA constructor (Christopher Corley, #174)
  • steps toward py3k compatibility (Lars Buitinck, #154)

0.8.9, 2013-12-26

  • use travis-ci for continuous integration
  • auto-optimize LDA asymmetric prior (Ben Trahan)
  • update for new word2vec binary format (Daren Race)
  • doc rendering fix (Dan Foreman-Mackey)
  • better LDA perplexity logging
  • fix Pyro thread leak in distributed algos (Brian Feeny)
  • optimizations in word2vec (Bryan Rink)
  • allow compressed input in LineSentence corpus (Eric Moyer)
  • upgrade ez_setup, doc improvements, minor fixes etc.

0.8.8 (aka "word2vec release"), 2013-11-03

  • python3 port by Parikshit Samant: https://github.com/samantp/gensimPy3
  • massive optimizations to word2vec (cython, BLAS, multithreading): ~20x-300x speedup
  • new word2vec functionality (thx to Ghassen Hamrouni, PR #124)
  • new CSV corpus class (thx to Zygmunt Zając)
  • corpus serialization checks to prevent overwriting (by Ian Langmore, PR #125)
  • add context manager support for older Python<=2.6 for gzip and bz2
  • added unittests for word2vec

0.8.7, 2013-09-18

  • initial version of word2vec, a neural network deep learning algo
  • make distributed gensim compatible with the new Pyro
  • allow merging dictionaries (by Florent Chandelier)
  • new design for the gensim website!
  • speed up handling of corner cases when returning top-n most similar
  • make Random Projections compatible with new scipy (andrewjOc360, PR #110)
  • allow "light" (faster) word lemmatization (by Karsten Jeschkies)
  • save/load directly from bzip2 files (by Luis Pedro Coelho, PR #101)
  • Blei corpus now tries harder to find its vocabulary file (by Luis Pedro Coelho, PR #100)
  • sparse vector elements can now be a list (was: only a 2-tuple)
  • simple_preprocess now optionally deaccents letters (ř/ĹĄ/Ăş=>r/s/u etc.)
  • better serialization of numpy corpora
  • print_topics() returns the topics, in addition to printing/logging
  • fixes for more robust Windows multiprocessing
  • lots of small fixes, data checks and documentation updates

0.8.6, 2012-09-15

  • added HashDictionary (by Homer Strong)
  • support for adding target classes in SVMlight format (by Corrado Monti)
  • fixed problems with global lemmatizer object when running in parallel on Windows
  • parallelization of Wikipedia processing + added script version that lemmatizes the input documents
  • added class method to initialize Dictionary from an existing corpus (by Marko Burjek)

0.8.5, 2012-07-22

  • improved performance of sharding (similarity queries)
  • better Wikipedia parsing (thx to Alejandro Weinstein and Lars Buitinck)
  • faster Porter stemmer (thx to Lars Buitinck)
  • several minor fixes (in HDP model thx to Greg Ver Steeg)
  • improvements to documentation

0.8.4, 2012-03-09

  • better support for Pandas series input (thx to JT Bates)
  • a new corpus format: UCI bag-of-words (thx to Jonathan Esterhazy)
  • a new model, non-parametric bayes: HDP (thx to Jonathan Esterhazy; based on Chong Wang's code)
  • improved support for new scipy versions (thx to Skipper Seabold)
  • lemmatizer support for wikipedia parsing (via the pattern python package)
  • extended the lemmatizer for multi-core processing, to improve its performance

0.8.3, 2011-12-02

  • fixed Similarity sharding bug (issue #65, thx to Paul Rudin)
  • improved LDA code (clarity & memory footprint)
  • optimized efficiency of Similarity sharding

0.8.2, 2011-10-31

  • improved gensim landing page
  • improved accuracy of SVD (Latent Semantic Analysis) (thx to Mark Tygert)
  • changed interpretation of LDA topics: github issue #57
  • took out similarity server code introduced in 0.8.1 (will become a separate project)
  • started using tox for testing
  • + several smaller fixes and optimizations

0.8.1, 2011-10-10

  • transactional similarity server: see docs/simserver.html
  • website moved from university hosting to radimrehurek.com
  • much improved speed of lsi[corpus] transformation:
  • accuracy tests of incremental svd: test/svderror.py and http://groups.google.com/group/gensim/browsethread/thread/4b605b72f8062770
  • further improvements to memory-efficiency of LDA and LSA
  • improved wiki preprocessing (thx to Luca de Alfaro)
  • model.print_topics() debug fncs now support std output, in addition to logging (thx to Homer Strong)
  • several smaller fixes and improvements

0.8.0 (Armageddon), 2011-06-28

  • changed all variable and function names to comply with PEP8 (numTopics->num_topics): BREAKS BACKWARD COMPATIBILITY!
  • added support for similarity querying more documents at once (index[querydocuments] in addition to index[querydocument]; much faster)
  • rewrote Similarity so that it is more efficient and scalable (using disk-based mmap'ed shards)
  • simplified directory structure (src/gensim/ is now only gensim/)
  • several small fixes and optimizations

0.7.8, 2011-03-26

  • added corpora.IndexedCorpus, a base class for corpus serializers (thx to Dieter Plaetinck). This allows corpus formats that inherit from it (MmCorpus, SvmLightCorpus, BleiCorpus etc.) to retrieve individual documents by their id in O(1), e.g. corpus[14] returns document #14.
  • merged new code from the LarKC.eu team (corpora.textcorpus, models.logentropy_model, lots of unit tests etc.)
  • fixed a bug in lda[bow] transformation (was returning gamma distribution instead of theta). LDA model generation was not affected, only transforming new vectors.
  • several small fixes and documentation updates

0.7.7, 2011-02-13

  • new LDA implementation after Hoffman et al.: Online Learning for Latent Dirichlet Allocation
  • distributed LDA
  • updated LDA docs (wiki experiments, distributed tutorial)
  • matrixmarket header now uses capital 'M's: MatrixMarket. (AndrĂŠ Lynum reported than Matlab has trouble processing the lowercase version)
  • moved code to github
  • started gensim Google group

0.7.6, 2011-01-10

  • added workaround for a bug in numpy: pickling a fortran-order array (e.g. LSA model) and then loading it back and using it results in segfault (thx to Brian Merrel)
  • bundled a new version of ez_setup.py: old failed with Python2.6 when setuptools were missing (thx to Alan Salmoni).

0.7.5, 2010-11-03

  • further optimization to LSA; this is the version used in my NIPS workshop paper
  • got rid of SVDLIBC dependency (one-pass LSA now uses stochastic algo for base-base decompositions)

0.7.4

  • sped up Latent Dirichlet ~10x (through scipy.weave, optional)
  • finally, distributed LDA! scales almost linearly, but no tutorial yet. see the tutorial on distributed LSI, everything's completely analogous.
  • several minor fixes and improvements; one nasty bug fixed (lsi[corpus] didn't work; thx to Danilo Spinelli)

0.7.3

  • added stochastic SVD decomposition (faster than the current one-pass LSI algo, but needs two passes over the input corpus)
  • published gensim on mloss.org

0.7.2

  • added workaround for a numpy bug where SVD sometimes fails to converge for no good reason
  • changed content of gensims's PyPi title page
  • completed HTML tutorial on distributed LSA

0.7.1

  • fixed a bug in LSA that occurred when the number of features was smaller than the number of topics (thx to Richard Berendsen)

0.7.0

  • optimized vocabulary generation in gensim.corpora.dictionary (faster and less memory-intense)
  • MmCorpus accepts compressed input (file-like objects such as GzipFile, BZ2File; to save disk space)
  • changed sparse solver to SVDLIBC (sparsesvd on PyPi) for large document chunks
  • added distributed LSA, updated tutorials (still experimental though)
  • several minor bug fixes

0.6.0

  • added option for online LSI training (yay!). the transformation can now be used after any amount of training, and training can be continued at any time with more data.
  • optimized the tf-idf transformation, so that it is a strictly one-pass algorithm in all cases (thx to Brian Merrell).
  • fixed Windows-specific bug in handling binary files (thx to Sutee Sudprasert)
  • fixed 1-based feature counting bug in SVMlight format (thx to Richard Berendsen)
  • added 'Topic :: Text Processing :: Linguistic' to gensim's pypi classifiers
  • change of sphinx documentation css and layout

0.5.0

  • finished all tutorials, stable version

0.4.7

  • tutorial on transformations

0.4.6

  • added Random Projections (aka Random Indexing), as another transformation model.
  • several DML-CZ specific updates

0.4.5

  • updated documentation
  • further memory optimizations in SVD (LSI)

0.4.4

  • added missing test files to MANIFEST.in

0.4.3

  • documentation changes
  • added gensim reference to Wikipedia articles (SVD, LSI, LDA, TFIDF, ...)

0.4.2

  • finally, a tutorial!
  • similarity queries got their own package

0.4.1

  • pdf documentation
  • removed dependency on python2.5 (theoretically, gensim now runs on 2.6 and 2.7 as well).

0.4.0

  • support for python setup.py test
  • fixing package metadata
  • documentation clean-up

0.2.0

  • First version

- Python
Published by mpenkov about 5 years ago

gensim - 4.0.0

Changes

4.0.0, 2021-03-24

⚠️ Gensim 4.0 contains breaking API changes! See the Migration guide to update your existing Gensim 3.x code and models.

Gensim 4.0 is a major release with lots of performance & robustness improvements, and a new website.

Main highlights

a. Efficiency

| model | 3.8.3: wall time / peak RAM / throughput | 4.0.0: wall time / peak RAM / throughput |
|----------|------------|--------|
| fastText | 2.9h / 4.11 GB / 822k words/s | 2.3h / **1.26 GB** / 914k words/s |
| word2vec | 1.7h / 0.36 GB / 1685k words/s | **1.2h** / 0.33 GB / 1762k words/s |

In other words, fastText now needs 3x less RAM (and is faster); word2vec has 2x faster init (and needs less RAM, and is faster); detecting collocation phrases is 2x faster. ([4.0 benchmarks](https://github.com/RaRe-Technologies/gensim/issues/2887#issuecomment-711097334))

b. Robustness. We fixed a bunch of long-standing bugs by refactoring the internal code structure (see 🔴 Bug fixes below)

c. Simplified OOP model for easier model exports and integration with TensorFlow, PyTorch &co.

These improvements come to you transparently aka "for free", but see Migration guide for some changes that break the old Gensim 3.x API. Update your code accordingly.

  • Dropped a bunch of externally contributed modules and wrappers: summarization, pivoted TFIDF, Mallet…

    • Code quality was not up to our standards. Also there was no one to maintain these modules, answer user questions, support them.

    So rather than let them rot, we took the hard decision of removing these contributed modules from Gensim. If anyone's interested in maintaining them, please fork & publish into your own repo. They can live happily outside of Gensim.

  • Dropped Python 2. Gensim 4.0 is Py3.6+. Read our Python version support policy.

    • If you still need Python 2 for some reason, stay at Gensim 3.8.3.

  • A new Gensim website – finally! 🙃

So, a major clean-up release overall. We're happy with this tighter, leaner and faster Gensim.

This is the direction we'll keep going forward: less kitchen-sink of "latest academic algorithms", more focus on robust engineering, targetting concrete NLP & document similarity use-cases.

:+1: New features

:books: Tutorials and docs

:red_circle: Bug fixes

  • #2891: Fix fastText word-vectors with ngrams off, by @gojomo
  • #2907: Fix doc2vec crash for large sets of doc-vectors, by @gojomo
  • #2899: Fix similarity bug in NMSLIB indexer, by @piskvorky
  • #2899: Fix deprecation warnings in Annoy integration, by @piskvorky
  • #2901: Fix inheritance of WikiCorpus from TextCorpus, by @jenishah
  • #2940: Fix deprecations in SoftCosineSimilarity, by @Witiko
  • #2944: Fix save_facebook_model failure after update-vocab & other initialization streamlining, by @gojomo
  • #2846: Fix for Python 3.9/3.10: remove xml.etree.cElementTree, by @hugovk
  • #2973: phrases.export_phrases() doesn't yield all bigrams, by @piskvorky
  • #2942: Segfault when training doc2vec, by @gojomo
  • #3041: Fix RuntimeError in export_phrases (change defaultdict to dict), by @thalishsajeed
  • #3059: Fix race condition in FastText tests, by @sleepy-owl

:warning: Removed functionality & deprecations

🔮 Testing, CI, housekeeping

- Python
Published by mpenkov about 5 years ago

gensim - 4.0.0.rc1

4.0.0.rc1, 2021-03-19

⚠️ Gensim 4.0 contains breaking API changes! See the Migration guide to update your existing Gensim 3.x code and models.

Gensim 4.0 is a major release with lots of performance & robustness improvements and a new website.

Main highlights (see also 👍 Improvements below)

a. Efficiency

| model | 3.8.3: wall time / peak RAM / throughput | 4.0.0: wall time / peak RAM / throughput |
|----------|------------|--------|
| fastText | 2.9h / 4.11 GB / 822k words/s | 2.3h / **1.26 GB** / 914k words/s |
| word2vec | 1.7h / 0.36 GB / 1685k words/s | **1.2h** / 0.33 GB / 1762k words/s |

In other words, fastText now needs 3x less RAM (and is faster); word2vec has 2x faster init (and needs less RAM, and is faster); detecting collocation phrases is 2x faster. ([4.0 benchmarks](https://github.com/RaRe-Technologies/gensim/issues/2887#issuecomment-711097334))

b. Robustness. We fixed a bunch of long-standing bugs by refactoring the internal code structure (see 🔴 Bug fixes below)

c. Simplified OOP model for easier model exports and integration with TensorFlow, PyTorch &co.

These improvements come to you transparently aka "for free", but see Migration guide for some changes that break the old Gensim 3.x API. Update your code accordingly.

  • Dropped a bunch of externally contributed modules: summarization, pivoted TFIDF normalization, FIXME.

    • Code quality was not up to our standards. Also there was no one to maintain them, answer user questions, support these modules.

    So rather than let them rot, we took the hard decision of removing these contributed modules from Gensim. If anyone's interested in maintaining them please fork into your own repo, they can live happily outside of Gensim.

  • Dropped Python 2. Gensim 4.0 is Py3.6+. Read our Python version support policy.

    • If you still need Python 2 for some reason, stay at Gensim 3.8.3.

  • A new Gensim website – finally! 🙃

So, a major clean-up release overall. We're happy with this tighter, leaner and faster Gensim.

This is the direction we'll keep going forward: less kitchen-sink of "latest academic algorithms", more focus on robust engineering, targetting common concrete NLP & document similarity use-cases.

:star2: New Features

:red_circle: Bug fixes

:books: Tutorial and doc improvements

  • fix various documentation warnings (mpenkov, #3077)
  • Fix broken link in rundoc how-to (sezanzeb_, #2991)
  • Point WordEmbeddingSimilarityIndex documentation to gensim.similarities (Witiko, #3003)
  • Make the link to the Gensim 3.8.3 documentation dynamic (Witiko, #2996)

:warning: Removed functionality

🔮 Miscellaneous

- Python
Published by mpenkov about 5 years ago

gensim - 4.0.0beta

4.0.0beta, 2020-10-31

⚠️ Gensim 4.0 contains breaking API changes! See the Migration guide to update your existing Gensim 3.x code and models.

Main highlights

a. Efficiency

| model | 3.8.3<br>wall time / peak RAM / throughput | 4.0.0<br>wall time / peak RAM / throughput |
|----------|------------|--------|
| fastText | 2.9h / 4.11 GB / 822k words/s | 2.3h / **1.26 GB** / 914k words/s |
| word2vec | 1.7h / 0.36 GB / 1685k words/s | **1.2h** / 0.33 GB / 1762k words/s |

In other words, fastText now needs **3x less RAM** (and is faster); word2vec has **2x faster init** (and needs less RAM, and is faster); detecting collocation phrases is **2x faster**. [4.0 benchmarks](https://github.com/RaRe-Technologies/gensim/issues/2887#issuecomment-711097334).

b. Robustness. We fixed a bunch of long-standing bugs by refactoring the internal code structure (see 🔴 Bug fixes below)

c. Simplified OOP model for easier model exports and integration with TensorFlow, PyTorch &co.

These improvements come to you transparently aka "for free", but see Migration guide for some changes that break the old Gensim 3.x API. Update your code accordingly.

  • Dropped a bunch of externally contributed modules: summarization, pivoted TFIDF normalization, wrappers for 3rd party libraries: Mallet, scikit-learn, DTM model, Vowpal Wabbit, wordrank, varembed.

    • Why? Code quality was not up to our standards. Also there was no one to maintain them, answer user questions, support these modules and wrappers.

    So rather than let them rot, we took the hard decision of removing these contributed modules from Gensim. If anyone's interested in maintaining them please fork into your own repo, they can live happily outside of Gensim, linked to as "contributed" from Gensim docs.

  • Dropped Python 2. Gensim 4.0 is Py3.6+. Read our Python version support policy.

    • If you still need Python 2 for some reason, stay at Gensim 3.8.3.

  • A new Gensim website – finally! 🙃

So, a major clean-up release overall. We're happy with this tighter, leaner and faster Gensim.

This is the direction we'll keep going forward: less kitchen-sink of "latest academic fad", more focus on robust engineering, targetting common NLP & document similarity use-cases.

Why a pre-release?

This 4.0.0beta pre-release is for users who want the cutting edge performance and bug fixes. Plus users who want to help out, by testing and providing feedback: code, documentation, workflows… Please let us know on the mailing list!

Install the pre-release with:

bash pip install --pre --upgrade gensim

What will change between this pre-release and a "full" 4.0 release?

Check progress here.

:+1: Improvements

:books: Tutorials and docs

:red_circle: Bug fixes

  • #2891: Fix fastText word-vectors with ngrams off, by @gojomo
  • #2907: Fix doc2vec crash for large sets of doc-vectors, by @gojomo
  • #2899: Fix similarity bug in NMSLIB indexer, by @piskvorky
  • #2899: Fix deprecation warnings in Annoy integration, by @piskvorky
  • #2901: Fix inheritance of WikiCorpus from TextCorpus, by @jenishah
  • #2940; Fix deprecations in SoftCosineSimilarity, by @Witiko
  • #2944: Fix save_facebook_model failure after update-vocab & other initialization streamlining, by @gojomo
  • #2846: Fix for Python 3.9/3.10: remove xml.etree.cElementTree, by @hugovk
  • #2973: phrases.export_phrases() doesn't yield all bigrams
  • #2942: Segfault when training doc2vec

:warning: Removed functionality & deprecations

  • #6: No more binary wheels for x32 platforms, by menshikh-iv
  • #2899: Renamed overly broad similarities.index to the more appropriate similarities.annoy, by @piskvorky
  • #2958: Remove gensim.summarization subpackage, docs and test data, by @mpenkov
  • #2926: Rename num_words to topn in dtm_coherence, by @MeganStodel
  • #2937: Remove Keras dependency, by @piskvorky
  • Removed all code, methods, attributes and functions marked as deprecated in Gensim 3.8.3.

- Python
Published by mpenkov over 5 years ago

gensim -

:warning: 3.8.x will be the last gensim version to support Py2.7. Starting with 4.0.0, gensim will only support Py3.5 and above

3.8.3, 2020-05-03

This is primarily a bugfix release to bring back Py2.7 compatibility to gensim 3.8.

:red_circle: Bug fixes

  • Bring back Py27 support (PR #2812, @mpenkov)
  • Fix wrong version reported by setup.py (Issue #2796)
  • Fix missing C extensions (Issues #2794 and #2802)

:+1: Improvements

:books: Tutorial and doc improvements

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use loadfacebookvectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and loadfacebookmodel to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by mpenkov about 6 years ago

gensim -

3.8.2, 2020-04-10

:red_circle: Bug fixes

  • Pin smart_open version for compatibility with Py2.7

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use loadfacebookvectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and loadfacebookmodel to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by mpenkov about 6 years ago

gensim -

3.8.1, 2019-09-23

:red_circle: Bug fixes

  • Fix usage of basedir instead of BASEDIR in loadinfo in downloader. (movb, #2605)
  • Update the version of smartopen in the setup.py file (AMR-KELEG_, #2582)
  • Properly handle unicodeerrors arg parameter when loading a vocab file (wmtzk_, #2570)
  • Catch loading older TfidfModels without smartirs (bnomis, #2559)
  • Fix bug where a module import set up logging, pin doctools for Py2 (piskvorky, #2552)

:books: Tutorial and doc improvements

:+1: Improvements

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use loadfacebookvectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and loadfacebookmodel to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by mpenkov over 6 years ago

gensim -

3.8.0, 2019-07-08

:warning: 3.8.x will be the last Gensim version to support Py2.7. Starting with 4.0.0, Gensim will only support Py3.5 and above

:star2: New Features

  • Enable online training of Poincare models (koiizukag, #2505)
  • Make BM25 more scalable by adding support for generator inputs (saraswatmks, #2479)
  • Allow the Gensim dataset / pre-trained model downloader gensim.downloader to run offline, by introducing a local file cache (mpenkov, #2545)
  • Make the gensim.downloader target directory configurable (mpenkov, #2456)
  • Support fast kNN document similarity search using NMSLIB (masa3141, #2417)

:red_circle: Bug fixes

  • Fix smart_open deprecation warning globally (itayB, #2530)
  • Fix AppVeyor issues with Windows and Py2 (mpenkov, #2546)
  • Fix topn=0 versus topn=None bug in most_similar, accept topn of any integer type (Witiko, #2497)
  • Fix Python version check (charsyam, #2547)
  • Fix typo in FastText documentation (Guitaricet, #2518)
  • Fix "Market Matrix" to "Matrix Market" typo. (Shooter23, #2513)
  • Fix auto-generated hyperlinks in CHANGELOG.md (mpenkov, #2482)

:books: Tutorial and doc improvements

  • Generate documentation for the gensim.similarities.termsim module (Witiko, #2485)
  • Simplify the Support section in README (piskvorky, #2542)

:+1: Improvements

  • Pin sklearn version for Py2, because sklearn dropped py2 support (mpenkov, #2510)

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use loadfacebookvectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and loadfacebookmodel to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by mpenkov almost 7 years ago

gensim -

3.7.3, 2019-05-06

:red_circle: Bug fixes

:books: Tutorial and doc improvements

:+1: Improvements

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use loadfacebookvectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and loadfacebookmodel to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by mpenkov about 7 years ago

gensim -

3.7.2, 2019-04-06

:star2: New Features

  • gensim.models.fasttext.load_facebook_model function: load full model (slower, more CPU/memory intensive, supports training continuation) ```python

    from gensim.test.utils import datapath

    cappath = datapath("crime-and-punishment.bin") fbmodel = loadfacebookmodel(cap_path)

    'landlord' in fbmodel.wv.vocab # Word is out of vocabulary False oovterm = fb_model.wv['landlord']

    'landlady' in fbmodel.wv.vocab # Word is in the vocabulary True ivterm = fb_model.wv['landlady']

    newsent = [['lord', 'of', 'the', 'rings'], ['lord', 'of', 'the', 'flies']] fbmodel.buildvocab(newsent, update=True) fbmodel.train(sentences=newsent, totalexamples=len(newsent), epochs=5)
    ```

  • gensim.models.fasttext.load_facebook_vectors function: load embeddings only (faster, less CPU/memory usage, does not support training continuation) ```python

    fbkv = loadfacebookvectors(cap_path)

    'landlord' in fbkv.vocab # Word is out of vocabulary False oov_vector = fbkv['landlord']

    'landlady' in fbkv.vocab # Word is in the vocabulary True iv_vector = fbkv['landlady'] ```

:red_circle: Bug fixes

  • Fix unicode error when loading FastText vocabulary (@mpenkov, #2390)
  • Avoid division by zero in fasttextinner.pyx (@mpenkov_, #2404)
  • Avoid incorrect filename inference when loading model (@mpenkov, #2408)
  • Handle invalid unicode when loading native FastText models (@mpenkov, #2411)
  • Avoid divide by zero when calculating vectors for terms with no ngrams (@mpenkov, #2411)

:books: Tutorial and doc improvements

:+1: Improvements

:warning: Changes in FastText behavior

Out-of-vocab word handling

To achieve consistency with the reference implementation from Facebook, a FastText model will now always report any word, out-of-vocabulary or not, as being in the model, and always return some vector for any word looked-up. Specifically:

  1. 'any_word' in ft_model will always return True. Previously, it returned True only if the full word was in the vocabulary. (To test if a full word is in the known vocabulary, you can consult the wv.vocab property: 'any_word' in ft_model.wv.vocab will return False if the full word wasn't learned during model training.)
  2. ft_model['any_word'] will always return a vector. Previously, it raised KeyError for OOV words when the model had no vectors for any ngrams of the word.
  3. If no ngrams from the term are present in the model, or when no ngrams could be extracted from the term, a vector pointing to the origin will be returned. Previously, a vector of NaN (not a number) was returned as a consequence of a divide-by-zero problem.
  4. Models may use more more memory, or take longer for word-vector lookup, especially after training on smaller corpuses where the previous non-compliant behavior discarded some ngrams from consideration.

Loading models in Facebook .bin format

The gensim.models.FastText.load_fasttext_format function (deprecated) now loads the entire model contained in the .bin file, including the shallow neural network that enables training continuation. Loading this NN requires more CPU and RAM than previously required.

Since this function is deprecated, consider using one of its alternatives (see below).

Furthermore, you must now pass the full path to the file to load, including the file extension. Previously, if you specified a model path that ends with anything other than .bin, the code automatically appended .bin to the path before loading the model. This behavior was confusing, so we removed it.

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.FastText.load_fasttext_format: use loadfacebookvectors to load embeddings only (faster, less CPU/memory usage, does not support training continuation) and loadfacebookmodel to load full model (slower, more CPU/memory intensive, supports training continuation)
    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by mpenkov about 7 years ago

gensim -

3.7.1, 2019-01-31

:+1: Improvements

:red_circle: Bug fixes

  • Fix infinite diff in LdaModel.do_mstep (@horpto, #2344)
  • Fix backward compatibility issue: loading FastTextKeyedVectors using KeyedVectors (missing attribute compatible_hash) (@menshikh-iv, #2349)
  • Fix logging issue (conda-forge related) (@menshikh-iv, #2339)
  • Fix WordEmbeddingsKeyedVectors.most_similar (@Witiko, #2356)
  • Fix issues of flake8==3.7.1 (@horpto, #2365)

:books: Tutorial and doc improvements

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by menshikh-iv over 7 years ago

gensim -

3.7.0, 2019-01-18

:star2: New features

  • Fast Online NMF (@anotherbugmaster, #2007)

    • Benchmark wiki-english-20171001

    | Model | Perplexity | Coherence | L2 norm | Train time (minutes) | |-------|------------|-----------|---------|----------------------| | LDA | 4727.07 | -2.514 | 7.372 | 138 | | NMF | 975.74 | -2.814 | 7.265 | 73 | | NMF (with regularization) | 985.57 | -2.436 | 7.269 | 441 | - Simple to use (same interface as LdaModel) ```python from gensim.models.nmf import Nmf from gensim.corpora import Dictionary import gensim.downloader as api

    text8 = api.load('text8')

    dictionary = Dictionary(text8) dictionary.filter_extremes()

    corpus = [ dictionary.doc2bow(doc) for doc in text8 ]

    nmf = Nmf( corpus=corpus, numtopics=5, id2word=dictionary, chunksize=2000, passes=5, randomstate=42, )

    nmf.show_topics() """ [(0, '0.007"km" + 0.006"est" + 0.006"islands" + 0.004"league" + 0.004"rate" + 0.004"female" + 0.004"economy" + 0.003"male" + 0.003"team" + 0.003"elections"'), (1, '0.006"actor" + 0.006"player" + 0.004"bwv" + 0.004"writer" + 0.004"actress" + 0.004"singer" + 0.003"emperor" + 0.003"jewish" + 0.003"italian" + 0.003"prize"'), (2, '0.036"college" + 0.007"institute" + 0.004"jewish" + 0.004"universidad" + 0.003"engineering" + 0.003"colleges" + 0.003"connecticut" + 0.003"technical" + 0.003"jews" + 0.003"universities"'), (3, '0.016"import" + 0.008"insubstantial" + 0.007"y" + 0.006"soviet" + 0.004"energy" + 0.004"info" + 0.003"duplicate" + 0.003"function" + 0.003"z" + 0.003"jargon"'), (4, '0.005"software" + 0.004"games" + 0.004"windows" + 0.003"microsoft" + 0.003"films" + 0.003"apple" + 0.003"video" + 0.002"album" + 0.002"fiction" + 0.002"characters"')] """ ``` - See also: - NMF tutorial - Full NMF Benchmark

  • Massive improvement of FastText compatibilities (@mpenkov, #2313) ```python from gensim.models import FastText

    'cc.ru.300.bin' - Russian Facebook FT model trained on Common Crawl

    Can be downloaded from https://s3-us-west-1.amazonaws.com/fasttext-vectors/word-vectors-v2/cc.ru.300.bin.gz

    model = FastText.loadfasttextformat("cc.ru.300.bin")

    Fixed hash-function allow to produce same output as FB FastText & works correctly for non-latin languages (for example, Russian)

    assert "мяу" in m.wv.vocab # 'мяу' - vocab word model.wv.most_similar("мяу") """ [('Мяу', 0.6820122003555298), ('МЯУ', 0.6373013257980347), ('мяу-мяу', 0.593108594417572), ('кис-кис', 0.5899622440338135), ('гав', 0.5866007804870605), ('Кис-кис', 0.5798211097717285), ('Кис-кис-кис', 0.5742273330688477), ('Мяу-мяу', 0.5699705481529236), ('хрю-хрю', 0.5508339405059814), ('ав-ав', 0.5479759573936462)] """

    assert "котогород" not in m.wv.vocab # 'котогород' - out-of-vocab word model.wv.most_similar("котогород", topn=3) """ [('автогород', 0.5463314652442932), ('ТагилНовокузнецкНовомосковскНовороссийскНовосибирскНовотроицкНовочеркасскНовошахтинскНовый', 0.5423436164855957), ('областьНовосибирскБарабинскБердскБолотноеИскитимКарасукКаргатКуйбышевКупиноОбьТатарскТогучинЧерепаново', 0.5377570390701294)] """

    Now we load full model, for this reason, we can continue an training

    from gensim.test.utils import datapath from smartopen import smartopen

    with smart_open(datapath("crime-and-punishment.txt"), encoding="utf-8") as infile: # russian text corpus = [line.strip().split() for line in infile]

    model.train(corpus, total_examples=len(corpus), epochs=5) ```

  • Similarity search improvements (@Witiko, #2016)

    • Add similarity search using the Levenshtein distance in gensim.similarities.LevenshteinSimilarityIndex
    • Performance optimizations to gensim.similarities.SoftCosineSimilarity (full benchmark)

    | dictionary size | corpus size | speed | |-----------------|-------------|--------------:| | 1000 | 100 | 1.0× | | 1000 | 1000 | 53.4× | | 1000 | 100000 | 156784.8× | | 100000 | 100 | 3.8× | | 100000 | 1000 | 405.8× | | 100000 | 100000 | 66262.0× | - See updated soft-cosine tutorial for more information and usage examples

  • Add python3.7 support (@menshikh-iv, #2211)

:+1: Improvements

Optimizations
  • Reduce Phraser memory usage (drop frequencies) (@jenishah, #2208)
  • Reduce memory consumption of summarizer (@horpto, #2298)
  • Replace inline slow equivalent of meanabsolutedifference with fast (@horpto, #2284)
  • Reuse precalculated updated prior in ldamodel.update_dir_prior (@horpto, #2274)
  • Improve KeyedVector.wmdistance (@horpto, #2326)
  • Optimize remove_unreachable_nodes in gensim.summarization (@horpto, #2263)
  • Optimize mz_entropy from gensim.summarization (@horpto, #2267)
  • Improve filter_extremes methods in Dictionary and HashDictionary (@horpto, #2303)
Additions
Cleanup

:red_circle: Bug fixes

:books: Tutorial and doc improvements

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by menshikh-iv over 7 years ago

gensim -

3.6.0, 2018-09-20

:star2: New features

Blog post / Jupyter tutorial.

New training mode for *2Vec models (word2vec, doc2vec, fasttext) that allows model training to scale linearly with the number of cores (full GIL elimination). The result of our Google Summer of Code 2018 project by Dmitry Persiyanov.

Benchmark on the full English Wikipedia, Intel(R) Xeon(R) CPU @ 2.30GHz 32 cores (GCE cloud), MKL BLAS:

| Model | Queue-based version [sec] | File-based version [sec] | speed up | Accuracy (queue-based) | Accuracy (file-based) | |-------|------------|--------------------|----------|----------------|-----------------------| | Word2Vec | 9230 | 2437 | 3.79x | 0.754 (Âą 0.003) | 0.750 (Âą 0.001) | | Doc2Vec | 18264 | 2889 | 6.32x | 0.721 (Âą 0.002) | 0.683 (Âą 0.003) | | FastText | 16361 | 10625 | 1.54x | 0.642 (Âą 0.002) | 0.660 (Âą 0.001) |

Usage:

```python import gensim.downloader as api from multiprocessing import cpucount from gensim.utils import saveaslinesentence from gensim.test.utils import get_tmpfile from gensim.models import Word2Vec, Doc2Vec, FastText

# Convert any corpus to the needed format: 1 document per line, words delimited by " " corpus = api.load("text8") corpusfname = gettmpfile("text8-file-sentence.txt") saveaslinesentence(corpus, corpusfname)

# Choose num of cores that you want to use (let's use all, models scale linearly now!) numcores = cpucount()

# Train models using all cores w2vmodel = Word2Vec(corpusfile=corpusfname, workers=numcores) d2vmodel = Doc2Vec(corpusfile=corpusfname, workers=numcores) ftmodel = FastText(corpusfile=corpusfname, workers=numcores)

``` Read notebook tutorial with full description.

:+1: Improvements

:red_circle: Bug fixes

:books: Tutorial and doc improvements

  • Update docstring with new analogy evaluation method (@akutuzov, #2130)
  • Improve prune_at parameter description for gensim.corpora.Dictionary (@yxonic, #2128)
  • Fix default -> auto prior parameter in documentation for lda-related models (@Laubeee, #2156)
  • Use heading instead of bold style in gensim.models.translation_matrix (@nzw0301, #2164)
  • Fix quote of vocabulary from gensim.models.Word2Vec (@nzw0301, #2161)
  • Replace deprecated parameters with new in docstring of gensim.models.Doc2Vec (@xuhdev, #2165)
  • Fix formula in Mallet documentation (@Laubeee, #2186)
  • Fix minor semantic issue in docs for Phrases (@RunHorst, #2148)
  • Fix typo in documentation (@KenjiOhtsuka, #2157)
  • Additional documentation fixes (@piskvorky, #2121)

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by menshikh-iv over 7 years ago

gensim - Docs 💬

3.5.0, 2018-07-06

This release comprises a glorious 38 pull requests from 28 contributors. Most of the effort went into improving the documentation—hence the release code name "Docs 💬"!

Apart from the massive overhaul of all Gensim documentation (including docstring style and examples—you asked for it), we also managed to sneak in some new functionality and a number of bug fixes. As usual, see the notes below for a complete list, with links to pull requests for more details.

Huge thanks to all contributors! Nobody loves working on documentation. 3.5.0 is a result of several months of laborious, unglamorous, and sometimes invisible work. Enjoy!

:books: Documentation improvements

:star2: New features:

  • Add add_entity method to KeyedVectors to allow adding word vectors manually (@persiyanov, #1957)
  • Add inference for new unseen author to AuthorTopicModel (@Stamenov, #1766)
  • Add evaluate_word_analogies (will replace accuracy) method to KeyedVectors (@akutuzov, #1935)
  • Add Pivot Normalization to TfidfModel (@markroxor, #1780)

:+1: Improvements

:red_circle: Bug fixes:

  • Fix Doc2Vec.infer_vector + notebook cleanup (@gojomo, #2103)
  • Fix linear decay for learning rate in Doc2Vec.infer_vector (@umangv, #2063)
  • Fix negative sampling floating-point error for `gensim.models.Poincare (@jayantj, #1959)
  • Fix loading word2vec and doc2vec models saved using old Gensim versions (@manneshiva, #2012)
  • Fix SoftCosineSimilarity.get_similarities on corpora ssues/1955) (@Witiko, #1972)
  • Fix return dtype for matutils.unitvec according to input dtype (@o-P-o, #1992)
  • Fix passing empty dictionary to gensim.corpora.WikiCorpus (@steremma, #2042)
  • Fix bug in Similarity.query_shards in multiprocessing case (@bohea, #2044)
  • Fix SMART from TfidfModel for case when df == "n" (@PeteBleackley, #2021)
  • Fix OverflowError when loading a large term-document matrix in compiled MatrixMarket format (@arlenk, #2001)
  • Update rules for removing table markup from Wikipedia dumps (@chaitaliSaini, #1954)
  • Fix _is_single from Phrases for case when corpus is a NumPy array (@rmalouf, #1987)
  • Fix tests for EuclideanKeyedVectors.similarity_matrix (@Witiko, #1984)
  • Fix deprecated parameters in D2VTransformer and W2VTransformer(@MritunjayMohitesh, #1945)
  • Fix Doc2Vec.infer_vector after loading old Doc2Vec (gensim<=3.2)(@manneshiva, #1974)
  • Fix inheritance chain for load_word2vec_format (@DennisChen0307, #1968)
  • Update Keras version (avoid bug from keras==2.1.5) (@menshikh-iv, #1963)

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by menshikh-iv almost 8 years ago

gensim -

3.4.0, 2018-03-01

:star2: New features:

  • Massive optimizations of gensim.models.LdaModel: much faster training, using Cython. (@arlenk, #1767)

    • Training benchmark :boom:

    | dataset | old LDA [sec] | optimized LDA [sec] | speed up | |---------|---------------|---------------------|---------| | nytimes | 3473 | 1975 | 1.76x | | enron | 774 | 437 | 1.77x | - This change affects all models that depend on LdaModel, such as LdaMulticore, LdaSeqModel, AuthorTopicModel.

  • Huge speed-ups to corpus I/O with MmCorpus (Cython) (@arlenk, #1825)

    • File reading benchmark

    | dataset | file compressed? | old MmReader [sec] | optimized MmReader [sec] | speed up | |---------------|:-----------:|:------------:|:------------------:|:-------------:| | enron | no | 22.3 | 2.6 | 8.7x | | | yes | 37.3 | 14.4 | 2.6x | | nytimes | no | 419.3 | 49.2 | 8.5x | | | yes | 686.2 | 275.1 | 2.5x | | text8 | no | 25.4 | 2.5 | 10.1x | | | yes | 41.9 | 17.0 | 2.5x | - Overall, a 2.5x speedup for compressed .mm.gz input and 8.5x :fire::fire::fire: for uncompressed plaintext .mm.

  • Performance and memory optimization to gensim.models.FastText :rocket: (@jbaiter, #1916)

    • Benchmark (first 500,000 articles from English Wikipedia)

    | Metric | old FastText | optimized FastText | improvement | | -----------------------| -----------------| -------------------|-------------| | Training time (1 epoch) | 4823.4s (80.38 minutes) | 1873.6s (31.22 minutes) | 2.57x | | Training time (full) | 1h 26min 13s | 36min 43s | 2.35x | | Training words/sec | 72,781 | 187,366 | 2.57x | | Training peak memory | 5.2 GB | 3.7 GB | 1.4x | - Overall, a 2.5x speedup & memory usage reduced by 30%.

  • Implemented Soft Cosine Measure (@Witiko, #1827)

    | Technique | MAP score | Duration | |-----------|-----------|--------------| | softcossim| 45.99 | 1.24 sec | | wmd-relax | 44.48 | 12.22 sec | | cossim | 44.22 | 4.39 sec | | wmd-gensim| 44.08 | 98.29 sec | - Soft Cosine notebook with detailed description, examples & benchmarks - Related papers: - Soft Similarity and Soft Cosine Measure: Similarity of Features in Vector Space Model - SimBow at SemEval-2017 Task 3: Soft-Cosine Semantic Similarity between Questions for Community Question Answering - Vector Space Representations in IR

:+1: Improvements:

  • New method to show the Gensim installation parameters: python -m gensim.scripts.package_info --info. Use this when reporting problems, for easier debugging. Fix #1902 (@sharanry, #1903)
  • Added a flag to optionally skip network-related tests, to help maintainers avoid network issues with CI services (@menshikh-iv, #1930)
  • Added license field to setup.py, allowing the use of tools like pip-licenses (@nils-werner, #1909)

:red_circle: Bug fixes:

  • Fix Python 3 compatibility for gensim.corpora.UciCorpus.save_corpus (@darindf, #1875)
  • Add wv property to KeyedVectors for backward compatibility. Fix #1882 (@manneshiva, #1884)
  • Fix deprecation warning from inspect.getargspec. Fix #1878 (@aneesh-joshi, #1887)
  • Add LabeledSentence to gensim.models.doc2vec for backward compatibility. Fix #1886 (@manneshiva, #1891)
  • Fix empty output bug in Phrases (when using model[tokens] twice). Fix #1401 (@sj29-innovate, #1853)
  • Fix type problems for D2VTransformer.fit_transform. Fix #1834 (@Utkarsh-Mishra-CIC, #1845)
  • Fix datatype parameter for KeyedVectors.load_word2vec_format. Fix #1682 (@pushpankar, #1819)
  • Fix deprecated parameters in doc2vec-lee notebook (@TheFlash10, #1918)
  • Fix file-like closing bug in gensim.corpora.MmCorpus. Fix #1869 (@sj29-innovate, #1911)
  • Fix precision problem in test_similarities.py, no more FP fails. (@menshikh-iv, #1928)
  • Fix encoding in Lee corpus reader. (@menshikh-iv, #1931)
  • Fix OOV pairs counter in WordEmbeddingsKeyedVectors.evaluate_word_pairs. (@akutuzov, #1934)

:books: Tutorial and doc improvements:

:warning: Deprecations (will be removed in the next major release)

  • Remove

    • gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation)
    • gensim.examples
    • gensim.nosy
    • gensim.scripts.word2vec_standalone
    • gensim.scripts.make_wiki_lemma
    • gensim.scripts.make_wiki_online
    • gensim.scripts.make_wiki_online_lemma
    • gensim.scripts.make_wiki_online_nodebug
    • gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation)
    • "deprecated" functions and attributes

  • Move

    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by menshikh-iv over 8 years ago

gensim -

3.3.0, 2018-02-02

:star2: New features: * Re-designed all "*2vec" implementations (@manneshiva, #1777) - Modular organization of Word2Vec, Doc2Vec, FastText, etc ..., making it easier to add new models in the future and re-use code - Fully backward compatible (even with loading models stored by a previous Gensim version) - Detailed documentation for the *2vec refactoring project

  • Improve gensim.scripts.segment_wiki by retaining interwiki links. Fix #1712 (@steremma, PR #1839)

    • Optionally extract interlinks from Wikipedia pages (use the --include-interlinks option). This will output one additional JSON dict for each article: { "interlinks": { "article title 1": "interlink text 1", "article title 2": "interlink text 2", ... } }

    • Example: extract the Wikipedia graph with article links as edges, from a raw Wikipedia dump: bash python -m gensim.scripts.segment_wiki --include-interlinks --file ~/Downloads/enwiki-latest-pages-articles.xml.bz2 --output ~/Desktop/enwiki-latest.jsonl.gz

      • Read this field from the segment_wiki output:

      ```python import json from smartopen import smartopen

      with smart_open("enwiki-latest.jsonl.gz") as infile: for doc in infile: doc = json.loads(doc)

          src_node = doc['title']
    dst_nodes = doc['interlinks'].keys()

    print(u"Source node: {}".format(src_node))
    print(u"Destination nodes: {}".format(u", ".join(dst_nodes)))
    break

      """ OUTPUT:

      Source node: Anarchism Destination nodes: anarcha-feminist, Ivan Illich, Adolf Brand, Josiah Warren, will (philosophy), anarcha-feminism, Anarchism in Mexico, Lysander Spooner, English Civil War, G8, Sebastien Faure, Nihilist movement, Sébastien Faure, Left-wing politics, imamate, Pierre Joseph Proudhon, anarchist communism, Università popolare (Italian newspaper), 1848 Revolution, Synthesis anarchism, labour movement, anarchist communists, collectivist anarchism, polyamory, post-humanism, postcolonialism, anti war movement, State (polity), security culture, Catalan people, Stoicism, Progressive education, stateless society, Umberto I of Italy, German language, Anarchist schools of thought, NEFAC, Jacques Ellul, Spanish Communist Party, Crypto-anarchism, ruling class, non-violence, Platformist, The History of Sexuality, Revolutions of 1917–23, Federación Anarquista Ibérica, propaganda of the deed, William B. Greene, Platformism, mutually exclusive, Fraye Arbeter Shtime, Adolf Hitler, oxymoron, Paris Commune, Anarchism in Italy#Postwar years and today, Oranienburg, abstentionism, Free Society, Henry David Thoreau, privative alpha, George I of Greece, communards, Gustav Landauer, Lucifer the Lightbearer, Moses Harman, coercion, regicide, rationalist, Resistance during World War II, Christ (title), Bohemianism, individualism, Crass, black bloc, Spanish Revolution of 1936, Erich Mühsam, Empress Elisabeth of Austria, Free association (communism and anarchism), general strike, Francesc Ferrer i Guàrdia, Catalan anarchist pedagogue and free-thinker, veganarchism, Traditional knowledge, Japanese Anarchist Federation, Diogenes of Sinope, Hierarchy, sexual revolution, Naturism, Bavarian Soviet Republic, February Revolution, Eugene Varlin, Renaissance humanism, Mexican Liberal Party, Friedrich Engels, Fernando Tarrida del Mármol, Caliphate, Marxism, Jesus, John Cage, Umanita Nova, Anarcho-pacifism, Peter Kropotkin, Religious anarchism, Anselme Bellegarrigue, civilisation, moral obligation, hedonist, Free Territory (Ukraine), -ism, neo-liberalism, Austrian School, philosophy, freethought, Joseph Goebbels, Conservatism, anarchist economics, Cavalier, Maximilien de Robespierre, Comstockery, Dorothy Day, Anarchism in France, Fédération anarchiste, World Economic Forum, Amparo Poch y Gascón, Sex Pistols, women's rights, collectivisation, Taoism, common ownership, William Batchelder Greene, Collective farming, popular education, biphobia, targeted killings, Protestant Christianity, state socialism, Marie François Sadi Carnot, Stephen Pearl Andrews, World Trade Organization, Communist Party of Spain (main), Pluto Press, Levante, Spain, Alexander Berkman, Wilhelm Weitling, Kharijites, Bolshevik, Liberty (1881–1908), Anarchist Aragon, social democrats, Dielo Truda, Post-left anarchy, Age of Enlightenment, Blanquism, Walden, mutual aid (organization), Far-left politics, privative, revolutions of 1848, anarchism and nationalism, punk rock, Étienne de La Boétie, Max Stirner, Jacobin (politics), agriculture, anarchy, Confederacion General del Trabajo de España, toleration, reformism, International Anarchist Congress of Amsterdam, The Ego and Its Own, Ukraine, Civil Disobedience (Thoreau), Spanish Civil War, David Graeber, Anarchism and issues related to love and sex, James Guillaume, Insurrectionary anarchism, Political repression, International Workers' Association, Barcelona, Bulgaria, Voline, Zeno of Citium, anarcho-communists, organized religion, libertarianism, bisexuality, Ricardo Flores Magón, Henri Zisly, Eight-hour day, Freetown Christiania, heteronormativity, Mikhail Bakunin, Propagandaministerium, Ezra Heywood, individual reappropriation, Modern School (United States), archon, Confédération nationale du travail, socialist movement, History of Islam, Max Nettlau, Political Justice, Reichstag fire, Anti-Christianity, decentralised, Issues in anarchism#Communism, deschooling, Christian movement, squatter, Anarchism in Germany, Catalonia, Louise Michel, Solidarity Federation, What is Property?, European individualist anarchism, Pierre-Joseph Proudhon, Mexican Revolution, wikt:anarchism, Blackshirts, Jewish anarchism, Russian Civil War, property rights, anti-authoritarian, individual reclamation, propaganda by the deed, from each according to his ability, to each according to his need, Feminist movement, Confiscation, social anarchism, Anarchism in Russia, Daniel Guérin, Uruguayan Anarchist Federation, Anarcha-feminism, Enragés, Cynicism (philosophy), workers' council, The Word (free love), Allen Ginsberg, Campaign for Nuclear Disarmament, antimilitarism, Workers' self-management, Federación Obrera Regional Argentina, self-governance, free market, Carlos I of Portugal, Simon Critchley, Anti-clericalism, heterosexual, Layla AbdelRahim, Mexican Anarchist Federation, Anarchism and Marxism, October Revolution, Anti-nuclear movement, Joseph Déjacque, Bolsheviks, Luigi Fabbri, morality, Communist party, Sam Dolgoff, united front, Ammon Hennacy, social ecology, commune (intentional community), Oscar Wilde, French Revolution, egoist anarchism, Comintern, transphobia, anarchism without adjectives, social control, means of production, Michel Onfray, Anarchism in France#The Fourth Republic (1945–1958), syndicalism, Anarchism in Spain, Iberian Anarchist Federation, International of Anarchist Federations, Emma Goldman, Netherlands, anarchist free school, International Workingmen's Association, Queer anarchism, Cantonal Revolution, trade unionism, Karl Marx, LGBT community, humanism, Anti-fascism, Carrara, political philosophy, Anarcho-transhumanism, libertarian socialist, Russian Revolution (1917), Two Cheers for Anarchism: Six Easy Pieces on Autonomy, Dignity, and Meaningful Work and Play, Emile Armand, insurrectionary anarchism, individual, Zhuang Zhou, Free Territory, White movement, Greenwich Village, Virginia Bolten, transcendentalist, public choice theory, wikt:brigand, Issues in anarchism#Participation in statist democracy, free love, Mutualism (economic theory), Anarchist St. Imier International, censorship, federalist, 6 February 1934 crisis, biennio rosso, anti-clerical, centralism, Anarchism: A Documentary History of Libertarian Ideas, minarchism, James C. Scott, First International, homosexuality, political theology, spontaneous order, Oranienburg concentration camp, anarcho-communism, negative liberty, post-modernism, Anarchism in Italy, Leopold Kohr, union of egoists, counterculture, Miguel Gimenez Igualada, philosophical anarchism, International Libertarian Solidarity, homosexual, Counterculture of the 1960s, Errico Malatesta, strikebreaker, Workers' Party of Marxist Unification, Clifford Harper, Reification (fallacy), patriarchy, anarchist law, Apostle (Christian), market (economics), Summerhill School, positive liberty, socialism, feminism, Direct action, Melchor Rodríguez García, William Godwin, Nazi concentration camps, Synthesist anarchism, Margaret Anderson, Han Ryner, Federation of Organized Trades and Labor Unions, technology, Workers Solidarity Movement, Edmund Burke, Encyclopædia Britannica, state (polity), Herbert Read, Park Güell, utilitarian, far right leagues, Limited government, self-ownership, Pejorative, homophobia, Industrial Workers of the World, The Dispossessed, Hague Congress (1872), Stalinism, Reciprocity (cultural anthropology), Fernand Pelloutier, individualist anarchism in France, The False Principle of our Education, individualist anarchism, Pierre Monatte, Soviet Union, counter-economics, Rudolf Rocker, Anarchism and capitalism, Parma, Black Rose Books, lesbian, Arditi del Popolo, Emile Armand (1872–1962), who propounded the virtues of free love in the Parisian anarchist milieu of the early 20th century, collectivism, Development criticism, John Henry Mackay, Benoît Broutchoux, Illegalism, Laozi, feminist, Christiaan Cornelissen, Syndicalist Workers' Federation, anarcho-syndicalism, Andalusia, Renzo Novatore, trade union, autonomist marxism, dictatorship of the proletariat, Mujeres Libres, Voltairine de Cleyre, Post-anarchism, participatory economics, Confederación Nacional del Trabajo, Syncretic politics, direct democracy, Jean-Jacques Rousseau, Green anarchism, Surrealism, labour unions, A. S. Neill, christian anarchist, Bonnot Gang, Anti-capitalism, Anarchism in Brazil, simple living, enlightened self-interest, Confédération générale du travail, class conflict, International Workers' Day, Hébertists, Gerrard Winstanley, Francoism, anarcho-pacifist, Andrej Grubacic, individualist anarchist and social anarchist thinkers., April Carter, private property, penal colonies, Libertarian socialism, Camillo Berneri, Christian anarchism, transhumanism, Lucifer, the Light-Bearer, Edna St. Vincent Millay, unschooling, Leo Tolstoy, M. E. Lazarus, Spanish Anarchists, Buddhist anarchism, ideology, William McKinley, anarcho-primitivism, Francesc Pi i Margall, :Category:Anarchism by country, International Workers Association, Anarcho-capitalism, Lois Waisbrooker, wikt:Solidarity, Baja California, social revolution, Unione Sindacale Italiana, Lev Chernyi, Alex Comfort, Sonnenburg, Leon Czolgosz, Volin, utopian, Argentine Libertarian Federation, Nudism, Left-wing market anarchism, insurrection, definitional concerns in anarchist theory, infinitive, affinity group, World Trade Organization Ministerial Conference of 1999 protest activity, class struggle, nonviolence, John Zerzan, poststructuralist, Noam Chomsky, Second Fitna, Julian Beck, Philadelphes, League of Peace and Freedom, Fédération Anarchiste, Kronstadt rebellion, Cold War, André Breton, Silvio Gesell, libertarian anarchism, voluntary association, anti-globalisation movement, birth control, L. Susan Brown, anarcho-naturism, personal property, Roundhead, Harold Barclay, The Joy of Sex, Council communism, Lucía Sánchez Saornil, tyrannicide, Neopaganism, lois scélérates, Johann Most, Anarchist Catalonia, Albert Camus, Protests of 1968, Alexander II of Russia, Spain's economy, Federazione Anarchica Italiana, Cuba, German Revolution of 1918–1919, stirner, Property is theft, Situationist International, law and economics

      ```

  • Add support for SMART notation for TfidfModel. Fix #1785 (@markroxor, #1791)

    • Natural extension of TfidfModel to allow different weighting and normalization schemes ```python from gensim.corpora import Dictionary from gensim.models import TfidfModel import gensim.downloader as api

      data = api.load("text8") dct = Dictionary(data) corpus = [dct.doc2bow(line) for line in data]

      Train Tfidf model using the SMART notation, smartirs="ntc" where

      'n' - natural term frequency

      't' - idf document frequency

      'c' - cosine normalization

      More information about possible values available in documentation or https://nlp.stanford.edu/IR-book/html/htmledition/document-and-query-weighting-schemes-1.html

      model = TfidfModel(corpus, id2word=dct, smartirs="ntc") vectorized_corpus = list(model[corpus])

      ```

    • SMART Information Retrieval System (wiki)

  • Add CircleCI for building Gensim documentation. Fix #1807 (@menshikh-iv, #1822)

    • An easy way to preview the rendered documentation (especially, if don't use Linux)
      • Go to "Details" link of CircleCI in your PR, click on the "Artifacts" tab, choose the HTML file that you want to view; a new tab will open with the rendered HTML page
    • Integration with Github, to see the documentation directly from the pull request page

:redcircle: Bug fixes: * Fix import in `getmyip`. Fix #1771 (@darindf, #1772) * Fix tox.ini/setup.cfg configuration (@menshikh-iv, #1815) * Fix formula in gensim.summarization.bm25. Fix #1828 (@sj29-innovate, #1833) * Fix the train method of TranslationMatrix (@robotcator, #1838) * Fix positional params used for gensim.models.CoherenceModel in gensim.models.callbacks (@Alexjmsherman, #1823) * Fix parameter setting for FastText.train. Fix #1818 (@sj29-innovate, #1837) * Pin python2 explicitly for building documentation (@menshikh-iv, #1840) * Remove dispatcher deadlock for distributed LDA (@darindf, #1817) * Fix `scorefunctionfromLexicalEntailmentEvaluation`. Fix #1858 (@hachibaka, #1863) * Fix symmetrical case for hellinger distance. Fix #1854 (@caiyulun, #1860) * Remove wrong logging at import. Fix #1706 (@menshikh-iv, #1871)

:books: Tutorial and doc improvements: * Refactor documentation API Reference for gensim.summarization (@yurkai & @menshikh-iv, #1709) * Fix docstrings for gensim.similarities.index. Partial fix #1666 (@menshikh-iv, #1681) * Fix docstrings for gensim.models.translation_matrix (@KokuKUSIAKU & @menshikh-iv, #1806) * Fix docstrings for gensim.models.rpmodel (@jazzmuesli & @menshikh-iv, #1802) * Fix docstrings for gensim.utils (@kakshay21 & @menshikh-iv, #1797) * Fix docstrings for gensim.matutils (@Cheukting & @menshikh-iv, #1804) * Fix docstrings for gensim.models.logentropy_model (@minggli & @menshikh-iv, #1803) * Fix docstrings for gensim.models.normmodel (@AustenLamacraft & @menshikh-iv, #1805) * Refactor API reference gensim.topic_coherence. Fix #1669 (@CLearERR & @menshikh-iv, #1714) * Fix documentation for gensim.corpora.dictionary and gensim.corpora.hashdictionary. Partial fix #1671 (@CLearERR & @menshikh-iv, #1814) * Fix documentation for gensim.corpora. Partial fix #1671 (@anotherbugmaster & @menshikh-iv, #1729) * Update banner in doc pages (@piskvorky, #1865) * Fix errors in the doc2vec-lee notebook (@PeterHamilton, #1841) * Add wordnet mammal train file for Poincare notebook (@jayantj, #1781) * Update Poincare notebooks (#1774) (@jayantj, #1774) * Update contributing guide. Fix #1786 (@menshikh-iv, #1793) * Add model_to_dict one-liner to word2vec notebook. Fix #1269 (@kakshay21, #1776) * Add word embedding viz to word2vec notebook. Fix #1419 (@markroxor, #1800) * Fix description of sg parameter for gensim.models.FastText (@akutuzov, #1801) * Fix typo in doc2vec-IMDB. Fix #1788 (@apoorvaeternity, #1796) * Remove outdated bz2 examples from tutorials2 * Remove outdated bz2 + MmCorpus examples from tutorials (@menshikh-iv, #1867)

:+1: Improvements: * Refactor tests for gensim.corpora.WikiCorpus (@steremma, #1821)

:warning: Deprecations (will be removed in the next major release) * Remove - gensim.models.wrappers.fasttext (obsoleted by the new native gensim.models.fasttext implementation) - gensim.examples - gensim.nosy - gensim.scripts.word2vec_standalone - gensim.scripts.make_wiki_lemma - gensim.scripts.make_wiki_online - gensim.scripts.make_wiki_online_lemma - gensim.scripts.make_wiki_online_nodebug - gensim.scripts.make_wiki (all of these obsoleted by the new native gensim.scripts.segment_wiki implementation) - "deprecated" functions and attributes

  • Move
    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by menshikh-iv over 8 years ago

gensim - Christmas Come Early

3.2.0, 2017-12-09

:star2: New features:

  • New download API for corpora and pre-trained models (@chaitaliSaini & @menshikh-iv, #1705 & #1632 & #1492)

    • Download large NLP datasets in one line of Python, then use with memory-efficient data streaming: ```python import gensim.downloader as api

      for article in api.load("wiki-english-20171001"): print(article)

      ```

    • Don’t waste time searching for good word embeddings, use the curated ones: ```python import gensim.downloader as api

      model = api.load("glove-twitter-25") model.most_similar("engineer")

      [('specialist', 0.957542896270752),

      ('developer', 0.9548177123069763),

      ('administrator', 0.9432312846183777),

      ('consultant', 0.93915855884552),

      ('technician', 0.9368376135826111),

      ('analyst', 0.9342101216316223),

      ('architect', 0.9257484674453735),

      ('engineering', 0.9159940481185913),

      ('systems', 0.9123805165290833),

      ('consulting', 0.9112802147865295)]

      ```

    • Blog post introducing the API and design decisions.

    • Jupyter notebook with examples

  • New model: PoincarĂŠ embeddings (@jayantj, #1696 & #1700 & #1757 & #1734)

    • Embed a graph (taxonomy) in the same way as word2vec embeds words: ```python from gensim.models.poincare import PoincareRelations, PoincareModel from gensim.test.utils import datapath

      data = PoincareRelations(datapath('poincarehypernyms.tsv')) model = PoincareModel(data) model.kv.mostsimilar("cat.n.01")

      [('kangaroo.n.01', 0.010581353439700418),

      ('gib.n.02', 0.011171531439892076),

      ('striped_skunk.n.01', 0.012025106076442395),

      ('metatherian.n.01', 0.01246679759214648),

      ('mammal.n.01', 0.013281303506525968),

      ('marsupial.n.01', 0.013941330203709653)]

      ```

    • Tutorial on PoincarĂŠ embeddings (Jupyter notebook).

    • Model introduction and the journey of its implementation (blog post).

    • Original paper on arXiv.

  • Optimized FastText (@manneshiva, #1742)

    • New fast multithreaded implementation of FastText, natively in Python/Cython. Deprecates the existing wrapper for Facebook’s C++ implementation. ```python import gensim.downloader as api from gensim.models import FastText

    model = FastText(api.load("text8")) model.most_similar("cat")

    [('catnip', 0.8538144826889038),

    ('catwalk', 0.8136177062988281),

    ('catchy', 0.7828493118286133),

    ('caf', 0.7826495170593262),

    ('bobcat', 0.7745151519775391),

    ('tomcat', 0.7732658386230469),

    ('moat', 0.7728310823440552),

    ('caye', 0.7666271328926086),

    ('catv', 0.7651021480560303),

    ('caveat', 0.7643581628799438)]

    ```

  • Binary pre-compiled wheels for Windows, OSX and Linux (@menshikh-iv, MacPython/gensim-wheels/#7)

    • Users no longer need to have a C compiler for using the fast (Cythonized) version of word2vec, doc2vec, fasttext etc.
    • Faster Gensim pip installation

  • Added DeprecationWarnings to deprecated methods and parameters, with a clear schedule for removal.

:+1: Improvements: * Add Montemurro and Zanette's entropy based keyword extraction algorithm. Fix #665 (@PeteBleackley, #1738) * Fix flake8 E731, E402, refactor tests & sklearn API code. Partial fix #1644 (@horpto, #1689) * Reduce distribution size. Fix #1698 (@menshikh-iv, #1699) * Improve scan_vocab speed, build_vocab_from_freq method (@jodevak, #1695) * Improve segment_wiki script (@piskvorky, #1707) * Add custom dtype support for LdaModel. Partially fix #1576 (@xelez, #1656) * Add doc2idx method for gensim.corpora.Dictionary. Fix #1634 (@roopalgarg, #1720) * Add tox and pytest to gensim, integration with Travis and Appveyor. Fix #1613, #1644 (@menshikh-iv, #1721) * Add flag for hiding outdated data for gensim.downloader.info (@menshikh-iv, #1736) * Add reproducible order between Python versions for gensim.corpora.Dictionary (@formi23, #1715) * Update tox.ini, setup.cfg, README.md (@menshikh-iv, #1741) * Add optimized logsumexp for LdaModel (@arlenk, #1745)

:redcircle: Bug fixes: * Fix ranking formula in gensim.summarization.bm25. Fix #1718 (@souravsingh, #1726) * Fixed incompatibility in persistence for FastText wrapper. Fix #1642 (@chinmayapancholi13, #1723) * Fix `gensim.sklearnapibug withdocumentscolumns` parameter. Fix #1676 (@chinmayapancholi13, #1704) * Fix slowdown of CI, remove pytest-cov (@menshikh-iv, #1728) * Replace outdated packages in Dockerfile (@rbahumi, #1730) * Replace `numwordstotopninLdaMallet.showtopics`. Fix #1747 (@apoorvaeternity, #1749) * Fix os.rename from gensim.downloader when 'src' and 'dst' on different partitions (@anotherbugmaster, #1733) * Fix DeprecationWarning from logsumexp (@dreamgonfly, #1703) * Fix backward compatibility problem in Phrases.load. Fix #1751 (@alexgarel, #1758) * Fix `loadword2vecformatfromFastText`. Fix #1743 (@manneshiva, #1755) * Fix ipython kernel version in Dockerfile. Fix #1762 (@rbahumi, #1764) * Fix writing in `segmentwiki(__[@horpto](https://github.com/horpto)__, [#1763](https://github.com/RaRe-Technologies/gensim/pull/1763)) * Fix write method of file requires byte-like object insegmentwiki` (@horpto, #1750) * Fix incorrect vectors learned during online training for FastText. Fix #1752 (@manneshiva, #1756) * Fix dtype of `model.wv.syn0vocabon updatingvocabforFastText. Fix #1759 (__[@manneshiva](https://github.com/manneshiva)__, [#1760](https://github.com/RaRe-Technologies/gensim/pull/1760)) * Fix hashing-trick fromFastText.buildvocab`. Fix #1765 (@manneshiva, #1768) * Add explicit DeprecationWarning for all outdated stuff. Fix #1753 (@menshikh-iv, #1769) * Fix epsilon according to dtype in LdaModel (@menshikh-iv_, #1770)

:books: Tutorial and doc improvements: * Update perf numbers of segment_wiki (@piskvorky, #1708) * Update docstring for gensim.summarization.summarize. Fix #1575 (@fbarrios, #1702) * Refactor API Reference for gensim.parsing. Fix #1664 (@CLearERR, #1684) * Fix typos in doc2vec-wikipedia notebook (@youqad, #1727) * Fix PyPI long description rendering (@edigaryev, #1739) * Fix twitter badge src (@menshikh-iv) * Fix maillist badge color (@menshikh-iv)

:warning: Deprecations (will be removed in the next major release) * Remove - gensim.examples - gensim.nosy - gensim.scripts.word2vec_standalone - gensim.scripts.make_wiki_lemma - gensim.scripts.make_wiki_online - gensim.scripts.make_wiki_online_lemma - gensim.scripts.make_wiki_online_nodebug - gensim.scripts.make_wiki

  • Move
    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

- Python
Published by menshikh-iv over 8 years ago

gensim -

3.1.0, 2017-11-06

:star2: New features: * Massive optimizations to LSI model training (@isamaru, #1620 & #1622) - LSI model allows use of single precision (float32), to consume 40% less memory while being 40% faster. - LSI model can now also accept CSC matrix as input, for further memory and speed boost. - Overall, if your entire corpus fits in RAM: 3x faster LSI training (SVD) in 4x less memory!

```python
# just an example; the corpus stream is up to you
streaming_corpus = gensim.corpora.MmCorpus("my_tfidf_corpus.mm.gz")

# convert your corpus to a CSC sparse matrix (assumes the entire corpus fits in RAM)
in_memory_csc_matrix = gensim.matutils.corpus2csc(streaming_corpus, dtype=np.float32)

# then pass the CSC to LsiModel directly
model = LsiModel(corpus=in_memory_csc_matrix, num_topics=500, dtype=np.float32)
```

  • Even if you continue to use streaming corpora (your training dataset is too large for RAM), you should see significantly faster processing times and a lower memory footprint. In our experiments with a very large LSI model, we saw a drop from 29 GB peak RAM and 38 minutes (before) to 19 GB peak RAM and 26 minutes (now):

    python model = LsiModel(corpus=streaming_corpus, num_topics=500, dtype=np.float32)

  • Phrases allows to use common terms in bigrams. Before, if you are searching to reveal ngrams like car_with_driver and car_without_driver, you can either remove stop words before processing, but you will only find car_driver, or you won't find any of those forms (because they have three words, but also because high frequency of with will avoid them to be scored correctly), inspired by ES common grams token filter.

    ```python phrold = Phrases(corpus) phrnew = Phrases(corpus, common_terms=stopwords.words('en'))

    print(phrold[["we", "provide", "car", "with", "driver"]]) # ["we", "provide", "carwith", "driver"] print(phrnew[["we", "provide", "car", "with", "driver"]]) # ["we", "provide", "carwith_driver"] ```

  • CLI script for processing a raw Wikipedia dump (the xml.bz2 format provided by MediaWiki) to extract its articles in a plain text format. It extracts each article's title, section names and section content and saves them as json-line:

    bash python -m gensim.scripts.segment_wiki -f enwiki-latest-pages-articles.xml.bz2 | gzip > enwiki-latest-pages-articles.json.gz

    Processing the entire English Wikipedia dump (13.5 GB, link here) takes about 2.5 hours (i7-6700HQ, SSD).

    The output format is one article per line, serialized into JSON:

    python for line in smart_open('enwiki-latest-pages-articles.json.gz'): # read the file we just created article = json.loads(line) print("Article title: %s" % article['title']) for section_title, section_text in zip(article['section_titles'], article['section_texts']): print("Section title: %s" % section_title) print("Section text: %s" % section_text)

:+1: Improvements: * Speedup FastText tests (@horpto, #1686) * Add optimization for SlicedCorpus.__len__ (@horpto, #1679) * Make word_vec return immutable vector. Fix #1651 (@CLearERR, #1662) * Drop Win x32 support & add rolling builds (@menshikh-iv, #1652) * Fix scoring function in Phrases. Fix #1533, #1635 (@michaelwsherman, #1573) * Add configuration for flake8 to setup.cfg (@mcobzarenco, #1636) * Add build_vocab_from_freq to Word2Vec, speedup scan_vocab (@jodevak, #1599) * Add most_similar_to_given method for KeyedVectors (@TheMathMajor, #1582) * Add __getitem__ method to Sparse2Corpus to allow direct queries (@isamaru, #1621)

:redcircle: Bug fixes: * Add single core mode to CoherenceModel. Fix #1683 (@horpto, #1685) * Fix ResourceWarnings in tests. Partially fix #1519 (@horpto, #1660) * Fix DeprecationWarnings generated by deprecated assertEquals. Partial fix #1519 (@poornagurram, #1658) * Fix DeprecationWarnings for regex string literals. Fix #1646 (@franklsf95, #1649) * Fix pagerank algorithm. Fix #805 (@xelez, #1653) * Fix FastText inconsistent dtype. Fix #1637 (@mcobzarenco, #1638) * Fix `testfilenamefiltering` test (@nehaljwani_, #1647)

:books: Tutorial and doc improvements: * Fix code/docstring style (@menshikh-iv, #1650) * Update error message for supervised FastText. Fix #1498 (@ElSaico, #1645) * Add "DOI badge" to README. Fix #1610 (@dphov, #1639) * Remove duplicate annoy notebook. Fix #1415 (@Karamax, #1640) * Fix duplication and wrong markup in docs (@horpto, #1633) * Refactor dendrogram & topic network notebooks (@parulsethi, #1571) * Fix release badge (@menshikh-iv, #1631)

:warning: Deprecation part (will come into force in the next major release) * Remove - gensim.examples - gensim.nosy - gensim.scripts.word2vec_standalone - gensim.scripts.make_wiki_lemma - gensim.scripts.make_wiki_online - gensim.scripts.make_wiki_online_lemma - gensim.scripts.make_wiki_online_nodebug - gensim.scripts.make_wiki

  • Move
    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

Also, we'll create experimental subpackage for unstable models. Specific lists will be available in the next major release.

- Python
Published by menshikh-iv over 8 years ago

gensim -

3.0.1, 2017-10-12

:redcircle: Bug fixes: * Fix Keras import, speedup importing time. Fix #1614 (@menshikh-v, #1615) * Fix Sphinx warnings and retrieve all missing .rst (@anotherbugmaster and @menshikh-iv, #1612) * Fix logger message in lsidispatcher (@lorosanu, #1603)

:books: Tutorial and doc improvements: * Fix spelling (@jberkel, #1625)

:warning: Deprecation part (will come into force in the next release) * Remove - gensim.examples - gensim.nosy - gensim.scripts.word2vec_standalone - gensim.scripts.make_wiki_lemma - gensim.scripts.make_wiki_online - gensim.scripts.make_wiki_online_lemma - gensim.scripts.make_wiki_online_nodebug - gensim.scripts.make_wiki

  • Move
    • gensim.scripts.make_wikicorpus ➡ gensim.scripts.make_wiki.py
    • gensim.summarization ➡ gensim.models.summarization
    • gensim.topic_coherence ➡ gensim.models._coherence
    • gensim.utils ➡ gensim.utils.utils (old imports will continue to work)
    • gensim.parsing.* ➡ gensim.utils.text_utils

Also, we'll create experimental subpackage for unstable models. Specific lists will be available in the next release.

- Python
Published by menshikh-iv over 8 years ago

gensim - GSoC storm

3.0.0, 2017-09-27

:star2: New features: * Add unsupervised FastText to Gensim (@chinmayapancholi13, #1525) * Add sklearn API for gensim models (@chinmayapancholi13, #1462) * Add callback metrics for LdaModel and integration with Visdom (@parulsethi, #1399) * Add TranslationMatrix model (@robotcator, #1434) * Add word2vec-based coherence. Fix #1380 (@macks22, #1530)

:+1: Improvements: * Add 'diagonal' parameter for LdaModel.diff (@parulsethi, #1448) * Add 'score' function for SklLdaModel (@chinmayapancholi13, #1445) * Update sklearn API for gensim models (@chinmayapancholi13, #1473) [:warning: breaks backward compatibility] * Add CoherenceModel to LdaModel.top_topics. Fix #1128 (@macks22, #1427) * Add dendrogram viz for topics and JS metric (@parulsethi, #1484) * Add topic network viz (@parulsethi, #1536) * Replace viewitems to iteritems. Fix #1495 (@HodorTheCoder, #1508) * Fix Travis config and add style-checking for Ipython Notebooks. Fix #1518, #1520 (@menshikh-iv, #1522) * Remove mutable args from definitions. Fix #1561 (@zsef123, #1562) * Add Appveyour for all PRs. Fix #1565 (@menshikh-iv, #1565) * Refactor code by PEP8. Partially fix #1521 (@zsef123, #1550) * Refactor code by PEP8 with additional limitations. Fix #1521 (@menshikh-iv, #1569) * Update FastTextKeyedVectors.__contains__ (@ELind77, #1499) * Update WikiCorpus tokenization. Fix #1534 (@roopalgarg, #1537)

:redcircle: Bug fixes: * Remove round in LdaSeqModel.printtopic. Fix #1480 (@menshikh-iv, #1547) * Fix TextCorpus.samletext (@menshikh-iv, #1548) * Fix Mallet wrapper and tests for HDPTransform (@menshikh-iv, #1555) * Fix incorrect initialization ShardedCorpus with a generator. Fix #1511 (@karkkainenk1, #1512) * Add verification when summarizecorpus returns null. Fix #1531 (@fbarrios, #1570) * Fix doctag unicode problem. Fix 1543 (@englhardt, #1544) * Fix Translation Matrix (@robotcator, #1594) * Add trainable flag to KeyedVectors.getembeddinglayer. Fix #1557 (@zsef123, #1558)

:books: Tutorial and doc improvements: * Update exception text in TextCorpus.samletext. Partial fix #308 (@vlejd, #1444) * Remove extra filtertoken from tutorial (@VorontsovIE, #1502) * Update Doc2Vec-IMDB notebook (@pahdo, #1476) * Add Google Tag Manager for site (@yardos, #1556) * Update docstring explaining lack of multistream support in WikiCopus. Fix #1496 (@polm and @menshikh-iv, #1515) * Fix PathLineSentences docstring (@gojomo) * Fix typos from Translation Matrix notebook (@robotcator, #1598)

- Python
Published by menshikh-iv over 8 years ago

gensim - Docker image and integration with Sklearn

2.3.0, 2017-07-25

:star2: New features: * Add Dockerfile for gensim with external wrappers (@parulsethi, #1368) * Add sklearn wrapper for Word2Vec (@chinmayapancholi13, #1437) * Add loss function for Word2Vec. Fix #999 (@chinmayapancholi13, #1201) * Add sklearn wrapper for AuthorTopic model (@chinmayapancholi13, #1403)

:+1: Improvements: * Remove unittest2 (@souravsingh, #1490) * Add multiple scoring methods for Phrases. Partial fix #1363 (@michaelwsherman, #1464) * Add WordRank wrapper to Dockerfile (@parulsethi, #1460) * Add PathLineSentences. Fix #1364 (@michaelwsherman, #1423) * Add TextDirectoryCorpus and refactor TextCorpus. Fix #1387 (@macks22, #1459) * Add sparse input support with topn parameter in any2sparse. Fix #1294 (@manneshiva, #1321) * Add seed and length for sampletext. Partial fix #308 (@vlejd, #1422) * Add wordngram parameter to FastText (@fsonntag, #1432)

:redcircle: Bug fixes: * Fix fastText loading from .bin file. Fix #1236 (@prakhar2b, #1341) * Fix paths in WordRank and running gensim version in Dockerfile (@parulsethi, #1503) * Fix commit version for gensim in Dockerfile (@parulsethi, #1491) * Fix encoding problems with tests on windows. Fix #1441 (@menshikh-iv, #1469) * Fix parameters in scorecbowpair (@jmhessel, #1468) * Fix parameters in scoresentence_cbow (@jmhessel, #1467) * Fix TextDirectoryCorpus on windows (@macks22, #1463) * Fix gensim version in Dockerfile (@parulsethi, #1456) * Fix WordOccurenceAccumulator on windows. Fix #1441 (@macks22, #1449) * Fix scipy/numpy requirements (downgrade). Fix #1450 (@menshikh-iv, #1450)

:books: Tutorial and doc improvements: * Fix links and spaces in quick start guide (@iamsanten, #1500) * Fix error of ConcatedDoc2Vec in doc2vec-imdb notebook (@robocator, #1377) * Fix Sphinx warnings. Fix #1192 (@prerna135, #1442) * Fix typo in LdaModel.diff method (@parulsethi, #1461) * Add Tensorboard visualization for LDA (@parulsethi, #1396) * Update old and add new notebook with CoherenceModel (@macks22, #1431)

- Python
Published by menshikh-iv almost 9 years ago

gensim - Integration with Keras and Sklearn, LdaModel topic difference

2.2.0, 2017-06-21

:star2: New features: * Add sklearn wrapper for RpModel (@chinmayapancholi13, #1395) * Add sklearn wrappers for LdaModel and LsiModel (@chinmayapancholi13, #1398) * Add sklearn wrapper for LdaSeq (@chinmayapancholi13, #1405) * Add keras wrapper for Word2Vec model (@chinmayapancholi13, #1248) * Add LdaModel.diff method (@menshikh-iv, #1334) * Allow use of truncated Dictionary for coherence measures. Fix #1342 (@macks22, #1349)

:+1: Improvements: * Fix saveastext/loadastext for Dictionary (@vlejd, #1402) * Add sampling support for corpus. Fix #308 (@vlejd, #1408) * Add napoleon extension to sphinx (@rasto2211, #1411) * Add KeyedVectors support to AnnoyIndexer (@quole, #1318) * Add BaseSklearnWrapper (@chinmayapancholi13, #1383) * Replace numwords to topn in model for unification. Fix #1198 (@prakhar2b, #1200) * Rename outpath to outname & add logging for WordRank model. Fix #1310 (@parulsethi, #1332) * Remove multiple iterations of corpus in pbooleandocument (@danielchamberlain, #1325) * Fix codestyle in TfIdf (@piskvorky, #1313) * Fix warnings from Sphinx. Partial fix #1192 (@souravsingh, #1330) * Add testenv to setup.py (@menshikh-iv, #1336)

:redcircle: Bug fixes: * Add cleanup in annoy test (@prakhar2b, #1420) * Add cleanup in lda backprop test (@prakhar2b, #1417) * Fix out-of-vocab in FastText (@jayantj, #1409) * Add cleanup in WordRank test (@parulsethi, #1410) * Fix rest requirements in Travis. Partial fix #1393 (@ibrahimsharaf, @menshikh-iv, #1400) * Fix morfessor exception. Partial fix #1324 (@souravsingh, #1406) * Fix test for FastText (@prakhar2b, #1371) * Fix WikiCorpus (@alekol, #1333) * Fix backward incompatibility for LdaModel (@chinmayapancholi13, #1327) * Fix support for old and new FastText model format. Fix #1301 (@prakhar2b, #1319) * Fix wrapper tests. Fix #1323 (@shubhamjain74, #1359) * Update exportphrases method. Fix #794 (@toumorokoshi, #1362) * Fix sklearn exception in test (@souravsingh, #1350)

:books: Tutorial and doc improvements: * Fix incorrect link in tutorials (@aneesh-joshi, #1426) * Add notebook with sklearn wrapper examples (@chinmayapancholi13, #1428) * Replace absolute pathes to relative in notebooks (@vochicong, #1414) * Fix code-style in keras notebook (@chinmayapancholi13, #1394) * Replace absolute pathes to relative in notebooks (@vochicong, #1407) * Fix typo in quickstart guide (@vochicong, #1404) * Update docstring for WordRank. Fix #1384 (@parulsethi, #1378) * Update docstring for SkLdaModel (@chinmayapancholi13, #1382) * Update logic for updatetype in LdaModel (@chinmayapancholi13, #1389) * Update docstring for Doc2Vec (@jstol, #1379) * Fix docstring for KL-distance (@viciousstar, #1373) * Update CorporaandVectorSpaces tutorial (@charliejharrison, #1308) * Add visualization for difference between LdaModel (@menshikh-iv, #1374) * Fix punctuation & typo in changelog (@piskvorky, @menshikh-iv, #1366) * Fix PEP8 & typo in several PRs (@menshikh-iv, #1369) * Update docstrings connected with backward compability in for LdaModel (@chinmayapancholi13, #1365) * Update CorporaandVectorSpaces tutorial (@schuyler1d, #1360) * Fix typo in Doc2Vec doctsring (@fujiyuu75, #1356) * Update Annoy tutorial (@pmbaumgartner, #1355) * Update temp folder in tutorials (@yl2526, #1352) * Remove spaces after print in TopicsandTransformation tutorial (@gsimore, #1354) * Update Dictionary docstring (@oonska, #1347) * Add section headings in word2vec notebook (@MikeTheReader, #1348) * Fix broken urls in starter tutorials (@ka7eh, #1346) * Update quick start notebook (@yardsale8, #1345) * Fix typo in quick start notebook (@MikeTheReader, #1344) * Fix docstring in keyedvectors (@chinmayapancholi13, #1337)

- Python
Published by menshikh-iv almost 9 years ago

gensim - Doc2Vec visualisation

2.1.0, 2017-05-12

:star2: New features: * Add modified saveword2vecformat for Doc2Vec, to save document vectors. (@parulsethi, #1256)

:+1: Improvements: * Add automatic code style check limited only to the code modified in PR (@tmylk, #1287) * Replace logger.warn by logger.warning (@chinmayapancholi13, #1295) * Docs word2vec docstring improvement, deprecation labels (@shubhvachher, #1274) * Stop passing 'sentences' as parameter to Doc2Vec. Fix #511 (@gogokaradjov, #1306)

:red_circle: Bug fixes: * Allow indexing with np.int64 in doc2vec. Fix #1231 (@bogdanteleaga, #1254) * Update Doc2Vec docstring. Fix #1302 (@datapythonista, #1307) * Ignore rst and ipynb file in Travis flake8 validations (@datapythonista, #1309)

:books: Tutorial and doc improvements: * Update Tensorboard Doc2Vec notebook (@parulsethi, #1286) * Update Doc2Vec IMDB Notebook, replace codesc to smart_open (@robotcator, #1278) * Add explanation of size to Word2Vec Notebook (@jbcoe, #1305) * Add extra param to WordRank notebook. Fix #1276 (@parulsethi, #1300) * Update warning message in WordRank (@parulsethi, #1299)

- Python
Published by menshikh-iv about 9 years ago

gensim - 2.0.0, 2017-04-10

Breaking changes:

Any direct calls to method train() of Word2Vec/Doc2Vec now require an explicit epochs parameter and explicit estimate of corpus size. The most usual way to call train is vec_model.train(sentences, total_examples=self.corpus_count, epochs=self.iter) See the method documentation for more information.

  • Explicit epochs and corpus size in word2vec train(). (@gojomo, @robotcator, #1139, #1237)

New features:

  • Add output word prediction in word2vec. Only for negative sampling scheme. See ipynb (@chinmayapancholi13,#1209)
  • scikit_learn wrapper for LSI Model in Gensim (@chinmayapancholi13,#1244)
  • Add the 'keeptokens' parameter to 'filterextremes'. (@toliwa,#1210)
  • Load FastText models with specified encoding (@jayantj,#1210)

Improvements: * Fix loading large FastText models on Mac. (@jaksmid,#1196) * Sklearn LDA wrapper now works in sklearn pipeline (@kris-singh,#1213) * glove2word2vec conversion script refactoring (@parulsethi,#1247) * Word2vec error message when update called before train . Fix #1162 (@hemavakade,#1205) * Allow training if model is not modified by "minimizemodel". Add deprecation warning. (@chinmayapancholi13,#1207) * Update the warning text when building vocab on a trained w2v model (@prakhar2b,#1190)

Bug fixes:

  • Fix word2vec reset_from bug in v1.0.1 Fix #1230. (@Kreiswolke,#1234)

  • Distributed LDA: checking the length of docs instead of the boolean value, plus int index conversion (@saparina ,#1191)

  • syn0lockf initialised with zero in intersectword2vec_format() (@KiddoZhu,#1267)

  • Fix wordrank maxiterdump calculation. Fix #1216 (@ajkl,#1217)

  • Make SgNegative test use sg (@shubhvachher ,#1252)

  • pep8/pycodestyle fixes for hanging indents in Summarization module (@SamriddhiJain ,#1202)

  • WordRank and Mallet wrappers single vs double quote issue in windows.(@prakhar2b,#1208)

  • Fix #824 : no corpus in init, but trim_rule in init (@prakhar2b ,#1186)

  • Hardcode version number. Fix #1138. ( @tmylk, #1138)

Tutorial and doc improvements:

  • Color dictionary according to topic notebook update (@bhargavvader, #1164)

  • Fix hdp show_topic/s docstring ( @parulsethi, #1264)

  • Add docstrings for word2vec.py forwarding functions ( @shubhvachher, #1251)

  • updated description for worker_loop function used in score function ( @chinmayapancholi13 , #1206)

- Python
Published by tmylk about 9 years ago

gensim - 1.0.1 Bug-fix release

  • Rebuild cumulative table on load. Fix #1180. (@tmylk,#1181)
  • mostsimilarcosmul bug fix (@dkim010, #1177)
  • Fix loading old word2vec models pre-1.0.0 (@jayantj, #1179)
  • Load utf-8 words in fasttext (@jayantj, #1176)

- Python
Published by tmylk about 9 years ago

gensim - 1.0.0 Author-Topic modelling

1.0.0, 2017-02-24

Deprecated methods:

In order to share word vector querying code between different training algos(Word2Vec, Fastext, WordRank, VarEmbed) we have separated storage and querying of word vectors into a separate class KeyedVectors.

Two methods and several attributes in word2vec class have been deprecated. The methods are load_word2vec_format and save_word2vec_format. The attributes are syn0norm, syn0, vocab, index2word . They have been moved to KeyedVectors class.

After upgrading to this release you might get exceptions about deprecated methods or missing attributes.

DeprecationWarning: Deprecated. Use model.wv.save_word2vec_format instead. AttributeError: 'Word2Vec' object has no attribute 'vocab'

To remove the exceptions, you should use KeyedVectors.load_word2vec_format instead of  Word2Vec.load_word2vec_format word2vec_model.wv.save_word2vec_format instead of  word2vec_model.save_word2vec_format model.wv.syn0norm instead of  model.syn0norm model.wv.syn0 instead of  model.syn0 model.wv.vocab instead of model.vocab model.wv.index2word instead of  model.index2word

Changelog of this release:

New features: - Add Author-topic modeling (@olavurmortensen,#893) - Add FastText word embedding wrapper (@Jayantj,#847) - Add WordRank word embedding wrapper (@parulsethi,#1066, #1125) - Add Varembed word embedding wrapper (@anmol01gulati, #1067)) - Add sklearn wrapper for LDAModel (@AadityaJ,#932)

Deprecated features: - Move load_word2vec_format and save_word2vec_format out of Word2Vec class to KeyedVectors (@tmylk,#1107) - Move properties syn0norm, syn0, vocab, index2word from Word2Vec class to KeyedVectors (@tmylk,#1147) - Remove support for Python 2.6, 3.3 and 3.4 (@tmylk,#1145)

Improvements: - Python 3.6 support (@tmylk #1077)
- Phrases and Phraser allow a generator corpus (ELind77 #1099) - Ignore DocvecsArray.doctagsyn0norm in save. Fix #789 (@accraze,#1053) - Fix bug in LsiModel that occurs when id2word is a Python 3 dictionary. (@cvangysel,#1103 - Fix broken link to paper in readme (@bhargavvader,#1101) - Lazy formatting in evaluatewordpairs (@akutuzov,#1084) - Deacc option to keywords pre-processing (@bhargavvader,#1076) - Generate Deprecated exception when using Word2Vec.loadword2vecformat (@tmylk, #1165) - Fix hdpmodel constructor docstring for printtopics (#1152) (@toliwa, #1152) - Default to perwordtopics=False in LDA getitem for performance (@menshikh-iv, #1154) - Fix bound computation in Author Topic models. (@olavurmortensen, #1156) - Write UTF-8 byte strings in tensorboard conversion (@tmylk,#1144) - Make toptopics and sparse2full compatible with numpy 1.12 strictly int idexing (@tmylk,#1146)

Tutorial and doc improvements: - Clarifying comment in iscorpus func in utils.py (@greninja,#1109) - Tutorial TopicsandTransformations fix markdown and add references (@lgmoneda,#1120) - Fix doc2vec-lee.ipynb results to match previous behavior (@bahbbc,#1119) - Remove Pattern lib dependency in News Classification tutorial (@luizcavalcanti,#1118) - CorporaandVectorSpaces tutorial text clarification (@lgmoneda,#1116) - Update Transformation and Topics link from quick start notebook (@mariana393,#1115) - Quick Start Text clarification and typo correction (@luizcavalcanti,#1114) - Fix typos in Author-topic tutorial (@Fil,#1102) - Address benchmark inconsistencies in Annoy tutorial (@droudy,#1113) - Add note about Annoy speed depending on numpy BLAS setup in annoytutorial.ipynb (@greninja,#1137) - Add documentation for WikiCorpus metadata. (@kirit93, #1163)

- Python
Published by tmylk over 9 years ago

gensim - 1.0.0RC2

1.0.0RC2, 2017-02-16

Deprecated methods:

In order to share word vector querying code between different training algos(Word2Vec, Fastext, WordRank, VarEmbed) we have separated storage and querying of word vectors into a separate class KeyedVectors.

Two methods and several attributes in word2vec class have been deprecated. The methods are load_word2vec_format and save_word2vec_format. The attributes are syn0norm, syn0, vocab, index2word . They have been moved to KeyedVectors class.

After upgrading to this release you might get exceptions about deprecated methods or missing attributes.

DeprecationWarning: Deprecated. Use model.wv.save_word2vec_format instead. AttributeError: 'Word2Vec' object has no attribute 'vocab'

To remove the exceptions, you should use KeyedVectors.load_word2vec_format instead of  Word2Vec.load_word2vec_format word2vec_model.wv.save_word2vec_format instead of  word2vec_model.save_word2vec_format model.wv.syn0norm instead of  model.syn0norm model.wv.syn0 instead of  model.syn0 model.wv.vocab instead of model.vocab model.wv.index2word instead of  model.index2word - Add note about Annoy speed depending on numpy BLAS setup in annoytutorial.ipynb (@greninja,#1137) - Remove direct access to properties moved to KeyedVectors (@tmylk,#1147) - Remove support for Python 2.6, 3.3 and 3.4 (@tmylk,#1145) - Write UTF-8 byte strings in tensorboard conversion (@tmylk,#1144) - Make top_topics and sparse2full compatible with numpy 1.12 strictly int idexing (@tmylk,#1146)

- Python
Published by tmylk over 9 years ago

gensim - Bug-fix for KeyedVector warnings in word2vec/doc2vec

0.13.4.1, 2017-01-04 - Disable direct access warnings on save and load of Word2vec/Doc2vec (@tmylk, #1072) - Making Default hs error explicit (@accraze, #1054) - Removed unnecessary numpy imports (@bhargavvader, #1065) - Utils and Matutils changes (@bhargavvader, #1062) - Tests for the evaluatewordpairs function (@akutuzov, #1061)

- Python
Published by tmylk over 9 years ago

gensim - KeyedVectors

Deprecation warning

After upgrading to this release you might see deprecation warnings like this:

WARNING:gensim.models.word2vec:direct access to syn0norm will not be supported in future gensim releases, please use model.wv.syn0norm

These warnings are correct and you are encouraged to change your Word2vec/Doc2vec code to use the new model.wv.syn0norm and model.wv.vocab fields instead of old direct access like model.syn0norm and model.vocab. The direct access will be deprecated in Feb 2017.

Specifically, you should use model.wv.syn0norm instead of  model.syn0norm model.wv.syn0 instead of  model.syn0 model.wv.vocab instead of model.vocab model.wv.index2word instead of  model.index2word

The reason for this deprecation is to separate word vectors from word2vec training. There are now new ways to get word vectors that don't involve training word2vec. We are adding capabilities to use word vectors trained in GloVe, FastText, WordRank, Tensorflow and Deeplearning4j word2vec. In order to have cleaner code and standard APIs for all word embeddings we extracted a KeyedVectors class and a word-vectors wv variable into the models.

0.13.4, 2016-12-22

Changelog: - Evaluation of word2vec models against semantic similarity datasets like SimLex-999 (#1047) (@akutuzov, #1047) - TensorBoard word embedding visualisation of Gensim Word2vec format (@loretoparisi, #1051) - Throw exception if load() is called on instance rather than the class in word2vec and doc2vec (@dus0x,(#889) - Loading and Saving LDA Models across Python 2 and 3. Fix #853 (@anmolgulati, #913, #1093) - Fix automatic learning of eta (prior over words) in LDA (@olavurmortensen, #1024). - eta should have dimensionality V (size of vocab) not K (number of topics). eta with shape K x V is still allowed, as the user may want to impose specific prior information to each topic. - eta is no longer allowed the "asymmetric" option. Asymmetric priors over words in general are fine (learned or user defined). - As a result, the eta update (update_eta) was simplified some. It also no longer logs eta when updated, because it is too large for that. - Unit tests were updated accordingly. The unit tests expect a different shape than before; some unit tests were redundant after the change; eta='asymmetric' now should raise an error. - Optimise showtopics to only call getlambda once. Fix #1006. (@bhargavvader, #1028) - HdpModel doc improvement. Inference and printtopics (@dsquareindia, #1029) - Removing Doc2Vec defaults so that it won't override Word2Vec defaults. Fix #795 (@markroxor, #929) Remove warning on gensim import "pattern not installed". Fix #1009 (@shashankg7, #1018) - Add deletetemporarytrainingdata() function to word2vec and doc2vec models. (@deepmipt-VladZhukov, #987) - New class KeyedVectors to store embedding separate from training code (@anmol01gulati and @droudy, #980) - Documentation improvements (@IrinaGoloshchapova, #1010, #1011) - LDA tutorial by Olavur, tips and tricks (@olavurmortensen, #779) - Add double quote in commmand line to run on Windows (@akarazeev, #1005) - Fix directory names in notebooks to be OS-independent (@mamamot, #1004) - Respect clipstart, clipend in mostsimilar. Fix #601. (@parulsethi, #994) - Replace Python sigmoid function with scipy in word2vec & doc2vec (@markroxor, #989) - WMD to return 0 instead of inf for sentences that contain a single word (@rbahumi, #986) - Pass all the params through the apply call in lda.getdocumenttopics(), test case to use the perwordtopics through the corpus in testldamodel (@parthoiiitm, #978) - Pyro annotations for lsi_worker (@markroxor, #968)

- Python
Published by tmylk over 9 years ago

gensim - Word2vec vocabulary expansion and documentation improvements

0.13.3, 2016-10-20 - Add vocabulary expansion feature to word2vec. (@isohyt, #900) - Tutorial: Reproducing Doc2vec paper result on wikipedia. (@isohyt, #654) - Add Save/Load interface to AnnoyIndexer for index persistence (@fortiema, #845) - Fixed issue #938,Creating a unified base class for all topic models. (@markroxor, #946) - breaking change in HdpTopicFormatter.show_topics - Add Phraser for Phrases optimization. ( @gojomo & @anujkhare , #837) - Fix issue #743, in word2vec's nsimilarity method if at least one empty list is passed ZeroDivisionError is raised (@pranay360, #883) - Change exportphrases in Phrases model. Fix issue #794 (@AadityaJ, #879) - bigram construction can now support multiple bigrams within one sentence - Fix issue #838, RuntimeWarning: overflow encountered in exp (@markroxor, #895) - Change some log messages to warnings as suggested in issue #828. (@rhnvrm, #884) - Fix issue #851, In summarizer.py, RunTimeError is raised if single sentence input is provided to avoid ZeroDivionError. (@metalaman, #887) - Fix issue #791, correct logic for iterating over SimilarityABC interface. (@MridulS, #839) - Fix RP model loading for large Fortran-order arrays (@piskvorky, #605) - Remove ShardedCorpus from init because of Theano dependency (@tmylk, #919) - Documentation improvements ( @dsquareindia & @tmylk, #914, #906 ) - Add Annoy memory-mapping example (@harshul1610, #899)

- Python
Published by tmylk over 9 years ago

gensim - Dynamic Topic Modelling in Python from Google Summer of Code. Breaking changes in hdp and dtm models. +15 changes.

0.13.2, 2016-08-19 - wordtopics has changed to wordtopics in ldamallet, and fixed issue #764. (@bhargavvader, #771) - assigning wordtopics value of wordtopics to keep backward compatibility, for now - topics, topn parameters changed to numtopics and numwords in showtopics() and printtopics()(@droudy, #755) - In hdpmodel and dtmmodel - NOT BACKWARDS COMPATIBLE! - Added randomstate parameter to LdaState initializer and checkrandomstate() (@droudy, #113) - Topic coherence update with `cuci,cnpmimeasures. LdaMallet, LdaVowpalWabbit support. Addtopicsparameter to coherencemodel. Can now provide tokenized topics to calculate coherence value. Faster backtracking. (@dsquareindia, [#750](https://github.com/RaRe-Technologies/gensim/pull/750), [#793](https://github.com/RaRe-Technologies/gensim/pull/793)) - Added a check for empty (no words) documents before starting to run the DTM wrapper if model = "fixed" is used (DIM model) as this causes the an error when such documents are reached in training. (@eickho, [#806](https://github.com/RaRe-Technologies/gensim/pull/806)) - New parameterslimit,datatype` for loadword2vecformat(); lockf for intersectword2vecformat (@gojomo, #817) - Changed `uselowercaseoption in word2vec accuracy tocaseinsensitive` to account for case variations in training vocabulary (@jayantj, #804 - Link to Doc2Vec on airline tweets example in tutorials page (@544895340 , #823) - Small error on Doc2vec notebook tutorial (@charlessutton, #816) - Bugfix: Full2sparse clipped to use abs value (@tmylk, #811) - WMD docstring: add tutorial link and query example (@tmylk, #813) - Annoy integration to speed word2vec and doc2vec similarity. Tutorial update (@droudy, #799,#792 ) - Add converter of LDA model between Mallet, Vowpal Wabit and gensim (@dsquareindia, #798, #766) - Distributed LDA in different network segments without broadcast (@menshikh-iv , #782) - Update CorporaandVectorSpaces.ipynb (@megansquire, #772) - DTM wrapper bug fixes caused by renaming numwords in #755 (@bhargavvader, #770) - Add LsiModel.docsprocessed attribute (@hobson, #763) - Dynamic Topic Modelling in Python. Google Summer of Code 2016 project. (@bhargavvader, #739, #831)

- Python
Published by tmylk almost 10 years ago

gensim - Topic Coherence

Initial release of Topic Coherence Cv and Umass. More work will be done here but external API will remain the same.

- Python
Published by tmylk almost 10 years ago

gensim - Tutorials reworked, Word Movers Distance

0.12.5, 2016

Tutorials migrated from website to ipynb (@j9chan, #721), (@jesford, #733, #725, 716) New doc2vec intro tutorial (@seanlaw, #730) Gensim Quick Start Tutorial (@andrewjlm, #727) Add exportphrases(sentences) to model Phrases (hanabi1224 #588) SparseMatrixSimilarity returns a sparse matrix if maintainsparsity is True (@davechallis, #590) added functionality for Topics of Words in document - i.e, dynamic topics. (@bhargavvader, #704) also included tutorial which explains new functionalities, and document word-topic coloring. Made normalization an explicit transformation. Added 'l1' norm support (@squareindia, #649) added term-topics API for most probable topic for word in vocab. (@bhargavvader, #706) buildvocab takes progressper parameter for smaller output (@zer0n, #624) Control whether to use lowercase for computing word2vec accuracy. (@alantian, #607) Easy import of GloVe vectors using Gensim (Manas Ranjan Kar, #625) Allow easy port of GloVe vectors into Gensim Standalone script with command line arguments, compatible with Python>=2.6 Usage: python -m gensim.scripts.glove2word2vec -i glovevectors.txt -o outputword2veccompatible.txt Add similarbyword() and similarby_vector() to word2vec (@isohyt, #381) Convenience method for similarity of two out of training sentences to doc2vec (@ellolo, #707) Dynamic Topic Modelling Tutorial updated with Dynamic Influence Model (@bhargavvader, #689) Added function to filter 'n' most frequent words from the dictionary (@abhinavchawla, #718) Raise warnings if vocab is single character elements and if alpha is increased in word2vec/doc2vec (@dsquareindia, #705) Tests for wikidump (@jonmcoe, #723) Mallet wrapper sparse format support (@RishabGoel, #664) Doc2vec pre-processing script translated from bash to Python (@andrewjlm, #720) Added Distance Metrics to matutils.pt (@bhargavvader, #656)

- Python
Published by tmylk almost 10 years ago

gensim - 0.13.0rc1 Tutorials reworked, Word Movers Distance

Changes

0.12.5, 2016 - Tutorials migrated from website to ipynb (@j9chan, #721), (@jesford, #733, #725, 716) - New doc2vec intro tutorial (@seanlaw, #730) - Gensim Quick Start Tutorial (@andrewjlm, #727) - Add exportphrases(sentences) to model Phrases (hanabi1224 #588) - SparseMatrixSimilarity returns a sparse matrix if `maintainsparsityis True (@davechallis, #590) - added functionality for Topics of Words in document - i.e, dynamic topics. (@bhargavvader, #704) - also included tutorial which explains new functionalities, and document word-topic coloring. - Made normalization an explicit transformation. Added 'l1' norm support (@squareindia, #649) - added term-topics API for most probable topic for word in vocab. (@bhargavvader, #706) - build_vocab takes progress_per parameter for smaller output (@zer0n, #624) - Control whether to use lowercase for computing word2vec accuracy. (@alantian, #607) - Easy import of GloVe vectors using Gensim (Manas Ranjan Kar, #625) - Allow easy port of GloVe vectors into Gensim - Standalone script with command line arguments, compatible with Python>=2.6 - Usage: python -m gensim.scripts.glove2word2vec -i glove_vectors.txt -o output_word2vec_compatible.txt - Addsimilarbyword()andsimilarbyvector()` to word2vec (@isohyt, #381) - Convenience method for similarity of two out of training sentences to doc2vec (@ellolo, #707) - Dynamic Topic Modelling Tutorial updated with Dynamic Influence Model (@bhargavvader, #689) - Added function to filter 'n' most frequent words from the dictionary (@abhinavchawla, #718) - Raise warnings if vocab is single character elements and if alpha is increased in word2vec/doc2vec (@dsquareindia, #705) - Tests for wikidump (@jonmcoe, #723) - Mallet wrapper sparse format support (@RishabGoel, #664) - Doc2vec pre-processing script translated from bash to Python (@andrewjlm, #720)

- Python
Published by tmylk almost 10 years ago

gensim - 0.12.4, 29/01/2016

  • Word2vec in line with original word2vec.c (Andrey Kutuzov, #538)
  • Same default values. See diff https://github.com/akutuzov/gensim/commit/6456cbcd75e6f8720451766ba31cc046b4463ae2
  • Standalone script with command line arguments matching those of original C tool. Usage ./word2vecstandalone.py -train data.txt -output trainedvec.txt -size 200 -window 2 -sample 1e-4
  • loadword2vecformat() performance (@svenkreiss, #555)
    • Remove init_sims() call for performance improvements when normalized vectors are not needed.
    • Remove norm_only parameter (API change). Call init_sims(replace=True) after the load_word2vec_format() call for the old norm_only=True behavior.
  • Better internal handling of job batching in word2vec (#535)
    • up to 300% speed up when training on very short documents (~tweets)
  • Word2vec allows non-strict unicode error handling (ignore or replace) (Gordon Mohr, #466)
  • Doc2Vec model.docvecs[key] now raises KeyError for unknown keys (Gordon Mohr, #520)
  • Fix DocvecsArray.index_to_doctag so most_similar() returns string doctags (Gordon Mohr, #560)
  • On-demand loading of the pattern library in utils.lemmatize (Jan Zikes, #461)
    • utils.HAS_PATTERN flag moved to utils.has_pattern()
  • Threadsafe Word2Vec/Doc2Vec finish-check to avoid hang/unending Word2Vec/Doc2Vec training (Gordon Mohr, #571)
  • Tuned TestWord2VecModel.test_cbow_hs() against random failures (Gordon Mohr, #531)
  • Prevent ZeroDivisionError when default_timer() indicate no elapsed time (Gordon Mohr, #518)
  • Forwards compatibility for NumPy > 1.10 (Matti Lyra, #494, #513)
    • LdaModel and LdaMulticore produce a large number of DeprecationWarnings from .inference() because the term ids in each chunk returned from utils.grouper are floats. This behaviour has been changed so that the term IDs are now ints.
    • utils.grouper returns a python list instead of a numpy array in .update() when LdaModel is called in non distributed mode
    • in distributed mode .update() will still call utils.grouper with as_numpy=True to save memory
    • LdaModel.update and LdaMulticore.update have a new keyword parameter chunksasnumpy=True/False (defaults to False) that allows controlling this behaviour

- Python
Published by tmylk over 10 years ago

gensim -

0.12.3rc1, 05/11/2015 - Make showtopics return value consistent across models (Christopher Corley, #448) - All models with the `showtopicsmethod should return a list of (topicnumber, topic)tuples, wheretopicis a list of (word, probability)tuples. - This is a breaking change that affects users of theLsiModel,LdaModel, andLdaMulticorethat may be reliant on the old tuple layout of (probability, word). - Mixed integer & string document-tags (keys to doc-vectors) will work (Gordon Mohr, #491) - DocvecsArray'sindex2doctaglist is renamed/reinterpreted asoffset2doctag -offset2doctagentries map todoctagsyn0indexes _after_ last plain-int doctag (if any) - (If using only string doctags,offset2doctagmay be interpreted same asindex2doctag`.) - New Tutorials on Dynamic Topic Modelling and Classification via Word2Vec (@arttii #471, @mataddy #500) - Auto-learning for the eta parameter on the LdaModel (Christopher Corley, #479) - Python 3.5 support - Speed improvements to keyword and summarisation methods (@erbas #441) - OSX wheels (#504) - Win build (#492)

- Python
Published by tmylk over 10 years ago

gensim -

0.12.3rc1, 05/11/2015 - Make showtopics return value consistent across models (Christopher Corley, #448) - All models with the `showtopicsmethod should return a list of (topicnumber, topic)tuples, wheretopicis a list of (word, probability)tuples. - This is a breaking change that affects users of theLsiModel,LdaModel, andLdaMulticorethat may be reliant on the old tuple layout of (probability, word). - Mixed integer & string document-tags (keys to doc-vectors) will work (Gordon Mohr, #491) - DocvecsArray'sindex2doctaglist is renamed/reinterpreted asoffset2doctag -offset2doctagentries map todoctagsyn0indexes _after_ last plain-int doctag (if any) - (If using only string doctags,offset2doctagmay be interpreted same asindex2doctag`.) - New Tutorials on Dynamic Topic Modelling and Classification via Word2Vec (@arttii #471, @mataddy #500) - Auto-learning for the eta parameter on the LdaModel (Christopher Corley, #479) - Python 3.5 support - Speed improvements to keyword and summarisation methods (@erbas #441) - OSX wheels (#504) - Win build (#492)

- Python
Published by tmylk over 10 years ago