Data

Tools

Indexes

Applications

Experiments

Open Source Science

multicore-tsne

Parallel t-SNE implementation with Python and Torch wrappers.

https://github.com/dmitryulyanov/multicore-tsne

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    1 of 18 committers (5.6%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.9%) to scientific vocabulary

Keywords

barnes-hut-tsne multicore py-bh-tsne tsne

Keywords from Contributors

docstring bioinformatics data-mining decision-trees distributed
Last synced: 10 months ago · JSON representation

Repository

Parallel t-SNE implementation with Python and Torch wrappers.

Basic Info
  • Host: GitHub
  • Owner: DmitryUlyanov
  • License: other
  • Language: C++
  • Default Branch: master
  • Homepage:
  • Size: 444 KB
Statistics
  • Stars: 1,913
  • Watchers: 42
  • Forks: 230
  • Open Issues: 40
  • Releases: 0
Topics
barnes-hut-tsne multicore py-bh-tsne tsne
Created over 9 years ago · Last pushed over 2 years ago
Metadata Files
Readme License

README.md

Multicore t-SNE Build Status

This is a multicore modification of Barnes-Hut t-SNE by L. Van der Maaten with Python CFFI-based wrappers. This code also works faster than sklearn.TSNE on 1 core (as of version 0.18).

What to expect

Barnes-Hut t-SNE is done in two steps.

  • First step: an efficient data structure for nearest neighbours search is built and used to compute probabilities. This can be done in parallel for each point in the dataset, this is why we can expect a good speed-up by using more cores.

  • Second step: the embedding is optimized using gradient descent. This part is essentially consecutive so we can only optimize within iteration. In fact some parts can be parallelized effectively, but not all of them a parallelized for now. That is why the second step speed-up will not be as significant as first step sepeed-up but there is still room for improvement.

So when can you benefit from parallelization? It is almost true, that the second step computation time is constant of D and depends mostly on N. The first part's time depends on D a lot, so for small D time(Step 1) << time(Step 2), for large D time(Step 1) >> time(Step 2). As we are only good at parallelizing step 1 we will benefit most when D is large enough (MNIST's D = 784 is large, D = 10 even for N=1000000 is not so much). I wrote multicore modification originally for Springleaf competition, where my data table was about 300000 x 3000 and only several days left till the end of the competition so any speed-up was handy.

Benchmark

1 core

Interestingly, this code beats other implementations. We compare to sklearn (Barnes-Hut of course), L. Van der Maaten's bhtsne, pybhtsne repo (cython wrapper for bhtsne with QuadTree). perplexity = 30, theta=0.5 for every run. In fact pybhtsne repo works at the same speed as this code when using more optimization flags for the compiler.

This is a benchmark for 70000x784 MNIST data:

| Method | Step 1 (sec) | Step 2 (sec) | | ---------------------------- |:---------------:| --------------:| | MulticoreTSNE(njobs=1) | 912 | 350 | | bhtsne | 4257 | 1233 | | pybh_tsne | 1232 | 367 | | sklearn(0.18) | ~5400 | ~20920 |

I did my best to find what is wrong with sklearn numbers, but it is the best benchmark I could do (you can find the test script in MulticoreTSNE/examples folder).

Multicore

This table shows a relative to 1 core speed-up when using n cores.

| n_jobs | Step 1 | Step 2 | | ------------- |:---------:| --------:| | 1 | 1x | 1x | | 2 | 1.54x | 1.05x | | 4 | 2.6x | 1.2x | | 8 | 5.6x | 1.65x |

How to use

Install

Directly from pypi

pip install MulticoreTSNE

From source

Make sure cmake is installed on your system, and you will also need a sensible C++ compiler, such as gcc or llvm-clang. On macOS, you can get both via homebrew.

To install the package, please do: bash git clone https://github.com/DmitryUlyanov/Multicore-TSNE.git cd Multicore-TSNE/ pip install .

Tested with python >= 3.6 (conda).

Run

You can use it as a near drop-in replacement for sklearn.manifold.TSNE.

```python from MulticoreTSNE import MulticoreTSNE as TSNE

tsne = TSNE(njobs=4) Y = tsne.fittransform(X) ```

Please refer to sklearn TSNE manual for parameters explanation.

This implementation n_components=2, which is the most common case (use Barnes-Hut t-SNE or sklearn otherwise). Also note that some parameters are there just for the sake of compatibility with sklearn and are otherwise ignored. See MulticoreTSNE class docstring for more info.

MNIST example

```python from sklearn.datasets import fetch_openml from MulticoreTSNE import MulticoreTSNE as TSNE from matplotlib import pyplot as plt

X, _ = fetchopenml( "mnist784", version=1, returnXy=True, asframe=False, parser="pandas" ) embeddings = TSNE(njobs=4).fittransform(X) visx = embeddings[:, 0] visy = embeddings[:, 1] plt.scatter(visx, visy, c=digits.target, cmap=plt.cm.getcmap("jet", 10), marker='.') plt.colorbar(ticks=range(10)) plt.clim(-0.5, 9.5) plt.show() ```

Test

You can test it on MNIST dataset with the following command:

bash python MulticoreTSNE/examples/test.py --n_jobs <n_jobs>

Note on jupyter use

To make the computation log visible in jupyter please install wurlitzer (pip install wurlitzer) and execute this line in any cell beforehand: %load_ext wurlitzer Memory leakages are possible if you interrupt the process. Should be OK if you let it run until the end.

License

Inherited from original repo's license.

Future work

  • Allow other types than double
  • Improve step 2 performance (possible)

Citation

Please cite this repository if it was useful for your research:

@misc{Ulyanov2016, author = {Ulyanov, Dmitry}, title = {Multicore-TSNE}, year = {2016}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/DmitryUlyanov/Multicore-TSNE}}, }

Of course, do not forget to cite L. Van der Maaten's paper

Owner

  • Name: Dmitry Ulyanov
  • Login: DmitryUlyanov
  • Kind: user
  • Location: Tel Aviv
  • Company: in3D_io

Co-Founder at Avaturn, in3D, Phd @ Skoltech

GitHub Events

Total
  • Watch event: 30
  • Issue comment event: 6
  • Fork event: 2
Last Year
  • Watch event: 30
  • Issue comment event: 6
  • Fork event: 2

Committers

Last synced: over 1 year ago

All Time
  • Total Commits: 99
  • Total Committers: 18
  • Avg Commits per committer: 5.5
  • Development Distribution Score (DDS): 0.576
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Dmitry Ulyanov d****u@g****m 42
Kernc k****e@g****m 31
falexwolf f****f@g****e 6
bjohnson b****n@g****m 2
Guenter U g****u 2
Milian Wolff m****f@k****m 2
cciccole c****a@g****m 2
jona-sassenhagen j****n@g****m 2
Thomas K Reynolds t****l@t****m 1
Erik Cederstrand e****k@c****k 1
Joshua Orvis j****s@g****m 1
Oleg o****7@g****m 1
Sebastian Bodenstein s****b@w****m 1
Sergiu 3****1 1
Shunya Ueta s****1@g****m 1
Sivasurya Santhanam s****n@g****m 1
Steffen Roecker s****r@g****m 1
thocevar t****r@f****i 1
Committer Domains (Top 20 + Academic)
fri.uni-lj.si: 1 wolfram.com: 1 cederstrand.dk: 1 tkreynol-mac01.ra.intel.com: 1 kdab.com: 1 gophronesis.com: 1 gmx.de: 1

Issues and Pull Requests

Last synced: 11 months ago

All Time
  • Total issues: 63
  • Total pull requests: 36
  • Average time to close issues: 7 months
  • Average time to close pull requests: 10 months
  • Total issue authors: 59
  • Total pull request authors: 26
  • Average comments per issue: 4.11
  • Average comments per pull request: 2.08
  • Merged pull requests: 24
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • LucaCappelletti94 (2)
  • ankane (2)
  • sg-s (2)
  • SumNeuron (2)
  • sovaa (1)
  • MonkeyChap (1)
  • MayukhSobo (1)
  • mkkkkk5 (1)
  • deepchatterjeevns (1)
  • redBirdTx (1)
  • neurOnur (1)
  • flde (1)
  • Quasimondo (1)
  • jreades (1)
  • leonexu (1)
Pull Request Authors
  • DmitryUlyanov (4)
  • falexwolf (4)
  • milianw (4)
  • guenteru (2)
  • bli25 (2)
  • luminosuslight (2)
  • ecederstrand (2)
  • tonetto (2)
  • kernc (2)
  • msakai (2)
  • alephthoughts (1)
  • jorvis (1)
  • chmodsss (1)
  • sroecker (1)
  • sbodenstein (1)
Top Labels
Issue Labels
Pull Request Labels

Packages

  • Total packages: 2
  • Total downloads:
    • pypi 937 last-month
  • Total dependent packages: 0
    (may contain duplicates)
  • Total dependent repositories: 19
    (may contain duplicates)
  • Total versions: 3
  • Total maintainers: 2
pypi.org: multicoretsne

Multicore version of t-SNE algorithm.

  • Versions: 2
  • Dependent Packages: 0
  • Dependent Repositories: 17
  • Downloads: 937 Last month
Rankings
Stargazers count: 1.6%
Downloads: 2.2%
Forks count: 3.4%
Dependent repos count: 3.5%
Average: 4.2%
Dependent packages count: 10.0%
Maintainers (2)
jorvis dulyanov
Last synced: 11 months ago
conda-forge.org: multicore-tsne

This is a multicore modification of Barnes-Hut t-SNE by L. Van der Maaten with python and Torch CFFI-based wrappers. This code also works faster than sklearn.TSNE on 1 core.

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 2
Rankings
Stargazers count: 9.5%
Forks count: 11.9%
Dependent repos count: 20.2%
Average: 23.3%
Dependent packages count: 51.6%
Last synced: 11 months ago

Dependencies

requirements.txt pypi
  • cffi *
  • cmake >=3.17.0
  • numpy *
setup.py pypi
  • cffi *
  • numpy *