https://github.com/moj-analytical-services/splink

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

Science Score: 49.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
✓
DOI references
Found 1 DOI reference(s) in README
○
Academic publication links
✓
Committers with academic emails
5 of 79 committers (6.3%) from academic institutions
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (14.2%) to scientific vocabulary

Keywords

data-matching data-science deduplicate-data deduplication duckdb em-algorithm entity-resolution fuzzy-matching record-linkage spark uk-gov-data-science

Keywords from Contributors

interactive embedded-database olap autograding interpretability projection interface sequences literature shellcodes

Last synced: 5 months ago · JSON representation

Repository

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

Basic Info

Host: GitHub
Owner: moj-analytical-services
License: mit
Language: Python
Default Branch: master
Homepage: https://moj-analytical-services.github.io/splink/
Size: 102 MB

Statistics

Stars: 1,693
Watchers: 17
Forks: 188
Open Issues: 211
Releases: 124

Topics

data-matching data-science deduplicate-data deduplication duckdb em-algorithm entity-resolution fuzzy-matching record-linkage spark uk-gov-data-science

Created about 6 years ago · Last pushed 6 months ago

Metadata Files

Readme Changelog Contributing License

README.md

Splink Logo

[!IMPORTANT] 🎉 Splink 4 has been released! Examples of new syntax are here and a release announcement is here.

Fast, accurate and scalable data linkage and deduplication

Splink is a Python package for probabilistic record linkage (entity resolution) that allows you to deduplicate and link records from datasets that lack unique identifiers.

It is used widely by within government, academia and the private sector - see use cases.

Key Features

⚡ Speed: Capable of linking a million records on a laptop in around a minute.
🎯 Accuracy: Support for term frequency adjustments and user-defined fuzzy matching logic.
🌐 Scalability: Execute linkage in Python (using DuckDB) or big-data backends like AWS Athena or Spark for 100+ million records.
🎓 Unsupervised Learning: No training data is required for model training.
📊 Interactive Outputs: A suite of interactive visualisations help users understand their model and diagnose problems.

Splink's linkage algorithm is based on Fellegi-Sunter's model of record linkage, with various customisations to improve accuracy.

What does Splink do?

Consider the following records that lack a unique person identifier:

tables showing what splink does

Splink predicts which rows link together:

tables showing what splink does

and clusters these links to produce an estimated person ID:

tables showing what splink does

What data does Splink work best with?

Splink performs best with input data containing multiple columns that are not highly correlated. For instance, if the entity type is persons, you may have columns for full name, date of birth, and city. If the entity type is companies, you could have columns for name, turnover, sector, and telephone number.

High correlation occurs when one column is highly predictable from another - for instance, city can be predicted from postcode. Correlation is particularly problematic if all of your input columns are highly correlated.

Splink is not designed for linking a single column containing a 'bag of words'. For example, a table with a single 'company name' column, and no other details.

Documentation

The homepage for the Splink documentation can be found here, including a tutorial and examples that can be run in the browser.

The specification of the Fellegi Sunter statistical model behind splink is similar as that used in the R fastLink package. Accompanying the fastLink package is an academic paper that describes this model. The Splink documentation site and a series of interactive articles also explores the theory behind Splink.

The Office for National Statistics have written a case study about using Splink to link 2021 Census data to itself.

Installation

Splink supports python 3.8+. To obtain the latest released version of splink you can install from PyPI using pip:

sh pip install splink

or, if you prefer, you can instead install splink using conda:

sh conda install -c conda-forge splink

Installing Splink for Specific Backends

For projects requiring specific backends, Splink offers optional installations for Spark, Athena, and PostgreSQL. These can be installed by appending the backend name in brackets to the pip install command: sh pip install 'splink[{backend}]'

Click here for backend-specific installation commands

#### Spark ```sh pip install 'splink[spark]' ``` #### Athena ```sh pip install 'splink[athena]' ``` #### PostgreSQL ```sh pip install 'splink[postgres]' ```

Quickstart

The following code demonstrates how to estimate the parameters of a deduplication model, use it to identify duplicate records, and then use clustering to generate an estimated unique person ID.

For more detailed tutorial, please see here.

```py import splink.comparisonlibrary as cl from splink import DuckDBAPI, Linker, SettingsCreator, blockon, splink_datasets

db_api = DuckDBAPI()

df = splinkdatasets.fake1000

settings = SettingsCreator( linktype="dedupeonly", comparisons=[ cl.JaroWinklerAtThresholds("firstname", [0.9, 0.7]), cl.JaroAtThresholds("surname", [0.9, 0.7]), cl.DateOfBirthComparison( "dob", inputisstring=True, datetimemetrics=["year", "month"], datetimethresholds=[1, 1], ), cl.ExactMatch("city").configure(termfrequencyadjustments=True), cl.EmailComparison("email"), ], blockingrulestogeneratepredictions=[ blockon("firstname"), blockon("surname"), ] )

linker = Linker(df, settings, db_api)

linker.training.estimateprobabilitytworandomrecordsmatch( [blockon("first_name", "surname")], recall=0.7, )

linker.training.estimateuusingrandomsampling(max_pairs=1e6)

linker.training.estimateparametersusingexpectationmaximisation( blockon("firstname", "surname") )

linker.training.estimateparametersusingexpectationmaximisation(block_on("dob"))

pairwisepredictions = linker.inference.predict(thresholdmatch_weight=-10)

clusters = linker.clustering.clusterpairwisepredictionsatthreshold( pairwise_predictions, 0.95 )

dfclusters = clusters.aspandas_dataframe(limit=5) ```

Videos

Support

To find the best place to ask a question, report a bug or get general advice, please refer to our Guide.

Awards

🥈 Civil Service Awards 2023: Best Use of Data, Science, and Technology - Runner up

🥇 Analysis in Government Awards 2022: People's Choice Award - Winner

🥈 Analysis in Government Awards 2022: Innovative Methods - Runner up

🥇 Analysis in Government Awards 2020: Innovative Methods - Winner

🥇 MoJ Data and Analytical Services Directorate (DASD) Awards 2020: Innovation and Impact - Winner

Citation

If you use Splink in your research, please cite as follows:

BibTeX @article{Linacre_Lindsay_Manassis_Slade_Hepworth_2022, title = {Splink: Free software for probabilistic record linkage at scale.}, author = {Linacre, Robin and Lindsay, Sam and Manassis, Theodore and Slade, Zoe and Hepworth, Tom and Kennedy, Ross and Bond, Andrew}, year = 2022, month = {Aug.}, journal = {International Journal of Population Data Science}, volume = 7, number = 3, doi = {10.23889/ijpds.v7i3.1794}, url = {https://ijpds.org/article/view/1794}, }

Acknowledgements

We are very grateful to ADR UK (Administrative Data Research UK) for providing the initial funding for this work as part of the Data First project.

We are extremely grateful to professors Katie Harron, James Doidge and Peter Christen for their expert advice and guidance in the development of Splink. We are also very grateful to colleagues at the UK's Office for National Statistics for their expert advice and peer review of this work. Any errors remain our own.

Related Repositories

While Splink is a standalone package, there are a number of repositories in the Splink ecosystem:

splink_scalaudfs contains the code to generate User Defined Functions in scala which are then callable in Spark.
splink_datasets contains datasets that can be installed automatically as a part of Splink through the In-build datasets functionality.
splinksyntheticdata contains code to generate synthetic data.

Owner

Name: MoJ Analytical Services
Login: moj-analytical-services
Kind: organization
Location: United Kingdom

Website: www.justice.gov.uk
Repositories: 208
Profile: https://github.com/moj-analytical-services

Committers

Last synced: 9 months ago

All Time

Total Commits: 7,501
Total Committers: 79
Avg Commits per committer: 94.949
Development Distribution Score (DDS): 0.626

Past Year

Commits: 916
Committers: 34
Avg Commits per committer: 26.941
Development Distribution Score (DDS): 0.305

Top Committers

Name	Email	Commits
Robin Linacre	r**e@d**k	2,809
ADBond	4****d	1,326
Ross Kennedy	r**y@d**k	928
Tom Hepworth	t**h@j**k	796
robin	a****s	760
zslade	z**e@g**m	256
Sam Lindsay	s**y@d**k	150
mamonu	t**m@m**m	77
Nick Crews	n**s@g**m	46
sama-ds	s**n@g**m	43
Zeb Burke-Conte	z**c@u**u	35
Tom Hepworth	4****J	33
aliceoleary0	1****0	26
aymonwuolanne	a**e@g**m	25
robert.whiffin	r**n@d**m	24
ericakane-moj	1****j	18
James Osmond	j**d@d**k	13
Robert Constable	r**9@g**m	10
afua-moj	a**i@j**k	9
Nicholas Erskine	n**5@g**m	8
sam lindsay	s**y@j**k	8
tylerdy	t**y@l**u	6
browo097302	1****2	5
Bobby W	b****y	4
Thomas-Hirsch	3****h	4
mihir-packmoose	m**r@p**m	4
Sam Kaufman	S**2@o**a	3
Benjamin	1****e	3
TommyBerry	t**y@d**k	3
cinnq346	c**6@g**m	3
and 49 more...

Committer Domains (Top 20 + Academic)

digital.justice.gov.uk: 4 justice.gov.uk: 2 uw.edu: 2 msn.com: 1 databricks.com: 1 dhsc.gov.uk: 1 justice.gsi.gov.uk: 1 long-slurm-sarchive-p0001.cluster.ihme.washington.edu: 1 packmoose.com: 1 ontario.ca: 1 gov.scot: 1 uniteus.com: 1 random-walk.dev: 1 rapidops.com: 1 trustscience.com: 1 ryte.ai: 1 long-slurm-sarchive-p0011.cluster.ihme.washington.edu: 1 example.comafua.kokayi: 1 cs.washington.edu: 1 dunelm.net: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 448
Total pull requests: 1,198
Average time to close issues: 4 months
Average time to close pull requests: 8 days
Total issue authors: 97
Total pull request authors: 59
Average comments per issue: 1.47
Average comments per pull request: 0.98
Merged pull requests: 985
Bot issues: 0
Bot pull requests: 31

Past Year

Issues: 92
Pull requests: 310
Average time to close issues: 23 days
Average time to close pull requests: 5 days
Issue authors: 39
Pull request authors: 23
Average comments per issue: 0.82
Average comments per pull request: 0.42
Merged pull requests: 267
Bot issues: 0
Bot pull requests: 5

View more stats

Top Authors

Issue Authors

ADBond (100)
RobinL (79)
RossKen (56)
NickCrews (22)
samnlindsay (22)
ThomasHepworth (19)
aymonwuolanne (10)
zslade (7)
samkodes (7)
medwar99 (6)
zmbc (5)
James-Osmond (4)
OlivierBinette (4)
lamaeldo (3)
sama-ds (3)

Pull Request Authors

RobinL (520)
ADBond (236)
RossKen (100)
ThomasHepworth (85)
zslade (38)
dependabot[bot] (31)
NickCrews (19)
zmbc (17)
aymonwuolanne (16)
samnlindsay (15)
browo097302 (7)
b-d-e (6)
tylerdy (6)
sama-ds (5)
afua-moj (4)

Top Labels

Issue Labels

enhancement (177) bug (97) documentation (45) good first issue (29) comparison levels (22) charts (20) splink4 (20) blocking (13) model training (13) continuous integration (12) clustering (12) dependencies (9) Interface/API improvement (9) check if still an issue (9) nice to have (9) profiling (9) user experience (8) spark (7) testing (7) postgres (7) sqlite (7) duckdb (6) performance (6) term frequency (6) settings (6) validation (5) refactoring (5) maintenance (5) graphs (5) caching (4)

Pull Request Labels

splink4 (85) dependencies (53) comparison levels (34) bug (29) documentation (28) continuous integration (22) testing (20) type hints (13) maintenance (12) spark (9) database api (8) cluster metrics (8) refactoring (7) validation (7) model training (6) merge_splink_4 (6) Interface/API improvement (6) postgres (6) enhancement (5) debug_mode (4) caching (4) python (4) clustering (4) dev (4) settings (3) github_actions (3) release (3) blocking (2) spellchecker (2) term frequency (2)

Packages

Total packages: 3
Total downloads:
- pypi 449,747 last-month
Total docker downloads: 100

Total dependent packages: 3
(may contain duplicates)
Total dependent repositories: 4
(may contain duplicates)
Total versions: 247
Total maintainers: 5

pypi.org: splink

Fast probabilistic data linkage at scale

Homepage: https://github.com/moj-analytical-services/splink
Documentation: https://splink.readthedocs.io/
License: MIT
Latest release: 4.0.8
published 9 months ago

Versions: 149
Dependent Packages: 3
Dependent Repositories: 4
Downloads: 449,747 Last month
Docker Downloads: 100

Rankings

Downloads: 1.1%

Stargazers count: 2.2%

Docker downloads count: 3.4%

Average: 3.9%

Forks count: 4.5%

Dependent packages count: 4.8%

Dependent repos count: 7.5%

Maintainers (5)

moj-analytical-services mamonu RobinL samnlindsay adbond

Last synced: 6 months ago

proxy.golang.org: github.com/moj-analytical-services/splink

Documentation: https://pkg.go.dev/github.com/moj-analytical-services/splink#section-documentation
License: mit
Latest release: v4.0.8+incompatible
published 9 months ago

Versions: 82
Dependent Packages: 0
Dependent Repositories: 0

Rankings

Dependent packages count: 5.6%

Average: 5.8%

Dependent repos count: 6.0%

Last synced: 6 months ago

conda-forge.org: splink

Homepage: https://github.com/moj-analytical-services/splink
License: MIT
Latest release: 1.0.6
published over 4 years ago

Versions: 16
Dependent Packages: 0
Dependent Repositories: 0

Rankings

Stargazers count: 18.8%

Forks count: 22.1%

Average: 31.5%

Dependent repos count: 34.0%

Dependent packages count: 51.2%

Last synced: 6 months ago

Dependencies

.github/workflows/autoblack.yml actions

actions/checkout v1 composite
actions/setup-python v1 composite

.github/workflows/documentation.yml actions

actions/checkout v3 composite
actions/setup-python v2 composite

.github/workflows/lint.yml actions

actions/checkout v2 composite
actions/setup-python v1 composite

.github/workflows/poetry_pypi_release.yml actions

actions/checkout v2 composite
actions/setup-python v3 composite

.github/workflows/pytest_36_compatibility.yml actions

actions/checkout v2 composite
gabrielfalcao/pyenv-action v9 composite

.github/workflows/pytest_benchmark_comment.yml actions

NejcZdovc/comment-pr v1 composite
actions/cache v2 composite
actions/checkout v2 composite
actions/setup-python v2 composite
snok/install-poetry v1 composite

.github/workflows/pytest_benchmark_commit.yml actions

actions/cache v2 composite
actions/checkout v2 composite
actions/setup-python v2 composite
snok/install-poetry v1 composite

.github/workflows/pytest_run_tests_with_cache.yml actions

actions/cache v2 composite
actions/checkout v2 composite
actions/setup-python v2 composite
snok/install-poetry v1 composite

.github/workflows/run_demos_examples.yml actions

actions/checkout v3 composite
actions/setup-python v2 composite

.github/workflows/run_demos_tutorials.yml actions

actions/checkout v3 composite
actions/setup-python v2 composite

benchmarking/requirements.txt pypi

altair ==4.2.0
duckdb ==0.3.2
jsonschema ==4.4.0
lzstring ==1.0.4
pandas ==1.4.1
pyspark ==3.1.2
pytest ==7.0.1
pytest-benchmark ==3.4.1
rapidfuzz ==2.0.2
sqlglot ==1.23.1
tabulate ==0.8.9

poetry.lock pypi

atomicwrites 1.4.1 develop
black 22.6.0 develop
click 8.1.3 develop
colorama 0.4.5 develop
dataclasses 0.8 develop
decorator 4.4.2 develop
flake8 4.0.1 develop
future 0.18.2 develop
iniconfig 1.1.1 develop
jarowinkler 1.1.2 develop
lzstring 1.0.4 develop
mccabe 0.6.1 develop
mypy-extensions 0.4.3 develop
networkx 2.5.1 develop
packaging 21.3 develop
pathspec 0.9.0 develop
platformdirs 2.5.2 develop
pluggy 1.0.0 develop
py 1.11.0 develop
py-cpuinfo 8.0.0 develop
py4j 0.10.9.3 develop
pyarrow 7.0.0 develop
pycodestyle 2.8.0 develop
pyflakes 2.4.0 develop
pyparsing 3.0.9 develop
pyspark 3.2.1 develop
pytest 7.1.2 develop
pytest-benchmark 3.4.1 develop
rapidfuzz 2.1.4 develop
tabulate 0.8.9 develop
tomli 2.0.1 develop
typed-ast 1.5.4 develop
altair 4.2.0
altair 4.1.0
attrs 21.4.0
duckdb 0.6.0
entrypoints 0.4
importlib-metadata 4.2.0
jinja2 3.1.2
jsonschema 3.2.0
markupsafe 2.1.1
numpy 1.21.1
pandas 1.1.5
pyrsistent 0.18.1
python-dateutil 2.8.2
pytz 2022.1
six 1.16.0
sqlglot 5.1.0
toolz 0.12.0
typing-extensions 4.3.0
zipp 3.8.1

pyproject.toml pypi

black ^22.1.0 develop
flake8 ^4.0.1 develop
lzstring 1.0.4 develop
networkx 2.5.1 develop
pyarrow ^7.0.0 develop
pyspark ^3.2.1 develop
pytest ^7.0 develop
pytest-benchmark ^3.4.1 develop
rapidfuzz ^2.0.3 develop
tabulate 0.8.9 develop
Jinja2 ^3.0.3
altair --- - !ruby/hash:ActiveSupport::HashWithIndifferentAccess version: "=4.1.0" python: "=3.6" - !ruby/hash:ActiveSupport::HashWithIndifferentAccess version: "^4.2.0" python: ">=3.7"
duckdb ^0.6.0
jsonschema >=3.2,<5.0
pandas ^1.0.0
python ^3.6
sqlglot 5.1.0