Data

Tools

Indexes

Applications

Experiments

Open Source Science

fancyimpute

Multivariate imputation and matrix completion algorithms implemented in Python

https://github.com/iskandr/fancyimpute

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ○
    codemeta.json file
  • ○
    .zenodo.json file
  • ✓
    DOI references
    Found 4 DOI reference(s) in README
  • ✓
    Academic publication links
    Links to: arxiv.org, ncbi.nlm.nih.gov, zenodo.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (12.2%) to scientific vocabulary
Last synced: 10 months ago · JSON representation

Repository

Multivariate imputation and matrix completion algorithms implemented in Python

Basic Info
  • Host: GitHub
  • Owner: iskandr
  • License: apache-2.0
  • Language: Python
  • Default Branch: master
  • Homepage:
  • Size: 286 KB
Statistics
  • Stars: 1,275
  • Watchers: 25
  • Forks: 180
  • Open Issues: 2
  • Releases: 5
Created over 10 years ago · Last pushed over 2 years ago
Metadata Files
Readme License

README.md

Build Status Coverage Status DOI

plot

A variety of matrix completion and imputation algorithms implemented in Python 3.6.

To install:

pip install fancyimpute

If you run into tensorflow problems and use anaconda, you can try to fix them with conda install cudatoolkit.

Important Caveats

(1) This project is in "bare maintenance" mode. That means we are not planning on adding more imputation algorithms or features (but might if we get inspired). Please do report bugs, and we'll try to fix them. Also, we are happy to take pull requests for more algorithms and/or features.

(2) IterativeImputer started its life as a fancyimpute original, but was then merged into scikit-learn and we deleted it from fancyimpute in favor of the better-tested sklearn version. As a convenience, you can still from fancyimpute import IterativeImputer, but under the hood it's just doing from sklearn.impute import IterativeImputer. That means if you update scikit-learn in the future, you may also change the behavior of IterativeImputer.

Usage

```python from fancyimpute import KNN, NuclearNormMinimization, SoftImpute, BiScaler

X is the complete data matrix

X_incomplete has the same values as X except a subset have been replace with NaN

Use 3 nearest rows which have a feature to fill in each row's missing features

Xfilledknn = KNN(k=3).fittransform(Xincomplete)

matrix completion using convex optimization to find low-rank solution

that still matches observed values. Slow!

Xfillednnm = NuclearNormMinimization().fittransform(Xincomplete)

Instead of solving the nuclear norm objective directly, instead

induce sparsity using singular value thresholding

Xincompletenormalized = BiScaler().fittransform(Xincomplete) Xfilledsoftimpute = SoftImpute().fittransform(Xincomplete_normalized)

print mean squared error for the imputation methods above

nnmmse = ((Xfillednnm[missingmask] - X[missingmask]) ** 2).mean() print("Nuclear norm minimization MSE: %f" % nnmmse)

softImputemse = ((Xfilledsoftimpute[missingmask] - X[missingmask]) ** 2).mean() print("SoftImpute MSE: %f" % softImputemse)

knnmse = ((Xfilledknn[missingmask] - X[missingmask]) ** 2).mean() print("knnImpute MSE: %f" % knnmse) ```

Algorithms

  • SimpleFill: Replaces missing entries with the mean or median of each column.

  • KNN: Nearest neighbor imputations which weights samples using the mean squared difference on features for which two rows both have observed data.

  • SoftImpute: Matrix completion by iterative soft thresholding of SVD decompositions. Inspired by the softImpute package for R, which is based on Spectral Regularization Algorithms for Learning Large Incomplete Matrices by Mazumder et. al.

  • IterativeImputer: A strategy for imputing missing values by modeling each feature with missing values as a function of other features in a round-robin fashion. A stub that links to scikit-learn's IterativeImputer.

  • IterativeSVD: Matrix completion by iterative low-rank SVD decomposition. Should be similar to SVDimpute from Missing value estimation methods for DNA microarrays by Troyanskaya et. al.

  • MatrixFactorization: Direct factorization of the incomplete matrix into low-rank U and V, with per-row and per-column biases, as well as a global bias. Solved by SGD in pure numpy.

  • NuclearNormMinimization: Simple implementation of Exact Matrix Completion via Convex Optimization by Emmanuel Candes and Benjamin Recht using cvxpy. Too slow for large matrices.

  • BiScaler: Iterative estimation of row/column means and standard deviations to get doubly normalized matrix. Not guaranteed to converge but works well in practice. Taken from Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares.

Citation

If you use fancyimpute in your academic publication, please cite it as follows: bibtex @software{fancyimpute, author = {Alex Rubinsteyn and Sergey Feldman}, title={fancyimpute: An Imputation Library for Python}, url = {https://github.com/iskandr/fancyimpute}, version = {0.7.0}, date = {2016}, }

Owner

  • Name: Alex Rubinsteyn
  • Login: iskandr
  • Kind: user
  • Company: University of North Carolina

Genes and vectors

GitHub Events

Total
  • Watch event: 27
  • Fork event: 2
Last Year
  • Watch event: 27
  • Fork event: 2

Committers

Last synced: about 1 year ago

All Time
  • Total Commits: 185
  • Total Committers: 11
  • Avg Commits per committer: 16.818
  • Development Distribution Score (DDS): 0.368
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Alex Rubinsteyn a****n@g****m 117
Sergey Feldman s****n@g****m 59
fritshermans p****t@f****l 1
Tim O'Donnell t****l@g****m 1
Sourabh Sandanshi s****i@g****m 1
P M 1****p 1
Karthik Ramanathan k****h@g****m 1
Johan Gustafsson j****n@g****m 1
Chelsea Zhang z****a@g****m 1
Brett Beaulieu-Jones b****j@g****m 1
Ahmet Can Ozbek o****n@g****m 1
Committer Domains (Top 20 + Academic)
fritshermans.nl: 1

Issues and Pull Requests

Last synced: 12 months ago

All Time
  • Total issues: 76
  • Total pull requests: 28
  • Average time to close issues: 3 months
  • Average time to close pull requests: 28 days
  • Total issue authors: 67
  • Total pull request authors: 13
  • Average comments per issue: 3.79
  • Average comments per pull request: 2.39
  • Merged pull requests: 22
  • Bot issues: 0
  • Bot pull requests: 1
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • sergeyf (4)
  • ambareeshsrja16 (2)
  • jamesmyatt (2)
  • pablo3p (2)
  • johannfaouzi (2)
  • Great1414 (2)
  • iskandr (2)
  • tangtangsiqi (1)
  • liyunrui (1)
  • irwinmier96 (1)
  • quancore (1)
  • arjunpuri7 (1)
  • redraven984 (1)
  • WoodruffA122 (1)
  • buddhadeb33 (1)
Pull Request Authors
  • sergeyf (14)
  • iskandr (3)
  • fritshermans (1)
  • ssandanshi (1)
  • YagmurGULEC (1)
  • gasper6 (1)
  • johongo (1)
  • drkarthi (1)
  • dependabot[bot] (1)
  • chelseaz (1)
  • kurtwheeler (1)
  • mabdelhack (1)
  • HCMY (1)
Top Labels
Issue Labels
Pull Request Labels
dependencies (1)

Packages

  • Total packages: 3
  • Total downloads:
    • pypi 78,156 last-month
  • Total docker downloads: 102
  • Total dependent packages: 7
    (may contain duplicates)
  • Total dependent repositories: 251
    (may contain duplicates)
  • Total versions: 34
  • Total maintainers: 3
pypi.org: fancyimpute

Matrix completion and feature imputation algorithms

  • Versions: 29
  • Dependent Packages: 6
  • Dependent Repositories: 251
  • Downloads: 78,156 Last month
  • Docker Downloads: 102
Rankings
Dependent repos count: 1.0%
Stargazers count: 1.9%
Average: 2.5%
Docker downloads count: 2.5%
Downloads: 2.7%
Dependent packages count: 3.2%
Forks count: 3.8%
Maintainers (3)
iskander openvax sergeyf
Last synced: 12 months ago
proxy.golang.org: github.com/iskandr/fancyimpute
  • Versions: 4
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Stargazers count: 2.1%
Forks count: 2.2%
Average: 3.9%
Dependent packages count: 5.4%
Dependent repos count: 5.7%
Last synced: 11 months ago
conda-forge.org: fancyimpute

![plot](https://user-images.strikinglycdn.com/res/hrscywv4p/image/upload/c_limit,fl_lossy,h_1440,w_720,f_auto,q_auto/174108/654579_364680.png) A variety of matrix completion and imputation algorithms implemented in Python 3.6. PyPI: [https://pypi.org/project/fancyimpute/](https://pypi.org/project/fancyimpute/)

  • Versions: 1
  • Dependent Packages: 1
  • Dependent Repositories: 0
Rankings
Stargazers count: 10.7%
Forks count: 12.5%
Average: 21.5%
Dependent packages count: 28.8%
Dependent repos count: 34.0%
Last synced: 11 months ago

Dependencies

requirements.txt pypi
  • cvxopt *
  • cvxpy *
  • knnimpute >=0.1.0
  • nose *
  • pytest *
  • scikit-learn >=0.24.2
setup.py pypi
  • cvxopt *
  • cvxpy *
  • knnimpute >=0.1.0
  • nose *
  • pytest *
  • scikit-learn >=0.24.2
  • used *