https://github.com/alleninstitute/ecephys_spike_sorting
Modules for processing extracellular electrophysiology data from Neuropixels probes
Science Score: 36.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
✓DOI references
Found 6 DOI reference(s) in README -
○Academic publication links
-
✓Committers with academic emails
5 of 8 committers (62.5%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.5%) to scientific vocabulary
Keywords
Repository
Modules for processing extracellular electrophysiology data from Neuropixels probes
Basic Info
Statistics
- Stars: 119
- Watchers: 15
- Forks: 101
- Open Issues: 20
- Releases: 0
Topics
Metadata Files
README.md
ecephys spike sorting
https://github.com/AllenInstitute/ecephysspikesorting
Modules for processing extracellular electrophysiology data from Neuropixels probes.
Overview
This repository contains code for running spike sorting pipelines for the Allen Brain Observatory. Public datasets that have been processed with ecephys_spike_sorting include Visual Coding - Neuropixels and Visual Behavior - Neuropixels. Electrophysiology that was spike sorted with this code has appeared in a number of publications, including:
Siegle, Jia et al. (2021) Survey of spiking in the mouse visual system reveals functional hierarchy.
Siegle, Ledochowitsch et al. (2021) Reconciling functional differences in populations of neurons recorded with two-photon imaging and electrophysiology.
Jia et al. (2022) Multi-regional module-based signal transmission in mouse visual cortex.
Compatibility
This code is designed to ingest data collected with the Open Ephys GUI. @jenniferColonell from HHMI Janelia Research Campus maintains a fork that is compatible with data recorded by SpikeGLX. For the spike sorting step, both versions rely on Kilosort 2 or 2.5. For more information on Kilosort, please read through the GitHub wiki.
Level of Support
This repository is no longer under development, and we recommend that new users base their spike sorting pipelines on SpikeInterface instead. We believe that even existing ecephys_spike_sorting users would benefit from migrating to SpikeInterface. The Allen Institute has already converted its spike sorting workflows to use SpikeInterface, which is actively maintained, works with a range of modern spike sorters, and includes up-to-date implementations of the most important pre- and post-processing methods. The SpikeInterface syntax needed to reproduce the functionality of ecephys_spike_sorting can be found in each module's README file.
To get started with SpikeInterface, we recommend reading through this tutorial on analyzing Neuropixels data.
Modules
The first three modules take data saved by the Open Ephys GUI and prepare it for spike sorting by Kilosort2. Following the spike-sorting step (using the kilosort_helper module), we clean up the outputs and calculate mean waveforms and quality metrics for each unit.
extractfromnpx (deprecated): Calls a binary executable that converts data from compressed NPX format into .dat files (continuous data) and .npy files (event data). The NPX format is no longer used by Open Ephys (or any other software), so this module can be skipped.
depth_estimation: Uses the LFP data to identify the surface channel, which is required by the median subtraction and kilosort modules.
median_subtraction: Calls a binary executable that removes the DC offset and common-mode noise from the AP band continuous file.
kilosort_helper: Generates config files for Kilosort and launches spike sorting via the Matlab engine.
kilosort_postprocessing: Removes putative double-counted spikes from Kilosort output.
noise_templates: Identifies noise units based on their waveform shape and ISI histogram.
mean_waveforms: Extracts mean waveforms from the raw data, given spike times and unit IDs. Also calculates metrics for each waveform.
quality_metrics: Calculates quality metrics for each unit to assess isolation and sorting quality.
(Not used) automerging: Automatically merges templates that belong to the same unit (included in case it's helpful to others).
Installation and Usage
These modules require Python 3.5+, and have been tested with Python 3.5, 3.6, and 3.7.
Three of the modules (extract_from_npx, median_subtraction, and kilosort_helper) have non-Python dependencies that will need to be installed prior to use.
We recommend using pipenv to run these modules. From the ecephys_spike_sorting top-level directory, run the following commands from a terminal:
Linux
shell
$ pip install --user pipenv
$ export PIPENV_VENV_IN_PROJECT=1
$ pipenv install
$ pipenv shell
(ecephys_spike_sorting) $ pip install .
You can now edit one of the processing scripts found in ecephys_spike_sorting/scripts and run via:
shell
(ecephys_spike_sorting) $ python ecephys_spike_sorting/scripts/batch_processing.py
See the scripts README file for more information on their usage.
To leave the pipenv virtual environment, simply type:
shell
(ecephys_spike_sorting) $ exit
macOS
If you don't have it already, install homebrew. Then, type:
shell
$ brew install pipenv
$ export PIPENV_VENV_IN_PROJECT=1
$ pipenv install
$ pipenv shell
(ecephys_spike_sorting) $ pip install .
You can now edit one of the processing scripts found in ecephys_spike_sorting/scripts and run via:
shell
(ecephys_spike_sorting) $ python ecephys_spike_sorting/scripts/batch_processing.py
See the scripts README file for more information on their usage.
To leave the pipenv virtual environment, simply type:
shell
(ecephys_spike_sorting) $ exit
Windows
shell
$ pip install --user pipenv
$ set PIPENV_VENV_IN_PROJECT=1
$ pipenv install
$ pipenv shell
(.venv) $ pip install .
Note: This will work in the standard Command Prompt, but the cmder console emulator has better compatibility with Python virtual environments.
You can now edit one of the processing scripts found in ecephys_spike_sorting\scripts and run via:
shell
(.venv) $ python ecephys_spike_sorting\scripts\batch_processing.py
See the scripts README file for more information on their usage.
To leave the pipenv virtual environment, simply type:
shell
(.venv) $ exit
Terms of Use
See Allen Institute Terms of Use
© 2019 Allen Institute for Brain Science
Owner
- Name: Allen Institute
- Login: AllenInstitute
- Kind: organization
- Location: Seattle, WA
- Website: https://alleninstitute.org
- Repositories: 184
- Profile: https://github.com/AllenInstitute
Please visit http://alleninstitute.github.io/ for more information.
GitHub Events
Total
- Issues event: 4
- Watch event: 8
- Issue comment event: 18
- Fork event: 1
Last Year
- Issues event: 4
- Watch event: 8
- Issue comment event: 18
- Fork event: 1
Committers
Last synced: 9 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Josh Siegle | j****s@a****g | 265 |
| Josh Siegle | j****h@a****g | 30 |
| maxym Myroshnychenko | m****s@g****m | 9 |
| Xiaoxuan Jia | x****j@a****g | 6 |
| Jingjie Li | j****1@u****k | 6 |
| NileGraddis | n****s@g****m | 3 |
| christopher mochizuki | m****8@g****m | 1 |
| Daniel Hähnke | d****e@t****e | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 7 months ago
All Time
- Total issues: 94
- Total pull requests: 59
- Average time to close issues: 2 months
- Average time to close pull requests: about 24 hours
- Total issue authors: 28
- Total pull request authors: 8
- Average comments per issue: 2.12
- Average comments per pull request: 0.66
- Merged pull requests: 43
- Bot issues: 0
- Bot pull requests: 14
Past Year
- Issues: 4
- Pull requests: 2
- Average time to close issues: about 19 hours
- Average time to close pull requests: 5 days
- Issue authors: 2
- Pull request authors: 1
- Average comments per issue: 4.25
- Average comments per pull request: 1.0
- Merged pull requests: 2
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- jsiegle (20)
- jiaxx (4)
- NileGraddis (4)
- JRicardo24 (3)
- LaurenzMuessig (2)
- OlyOlympic (2)
- ldclaar (2)
- plodocus (2)
- edwardyan95 (1)
- nsteinme (1)
- bartulem (1)
- lihao881230 (1)
- saharmanavi (1)
- WeissShahaf (1)
- DaniMagyar (1)
Pull Request Authors
- jsiegle (15)
- dependabot[bot] (8)
- greggheller (4)
- jingjie-li (4)
- mmyros (3)
- cagjony (1)
- mochic (1)
- plodocus (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 17 last-month
- Total dependent packages: 1
- Total dependent repositories: 0
- Total versions: 2
- Total maintainers: 1
pypi.org: forked-ecephys-spike-sorting
Tools for spike-sorting Allen Institute Neuropixels data
- Homepage: https://github.com/AllenInstitute/ecephys_spike_sorting
- Documentation: https://forked-ecephys-spike-sorting.readthedocs.io/
- License: other
-
Latest release: 0.1.1
published over 2 years ago
Rankings
Maintainers (1)
Dependencies
- aibs-portal-assets 0.0.24
- bumpversion * develop
- cookiecutter * develop
- coverage * develop
- pytest * develop
- ruamel.yaml * develop
- sphinx * develop
- sphinx-gallery * develop
- tox * develop
- GitPython *
- argschema ==1.17.5
- h5py *
- joblib *
- marshmallow ==2.19.2
- matplotlib *
- numpy *
- pandas *
- requests >=2.20.0
- scikit-learn *
- scipy *
- urllib3 >=1.24.2
- xarray *
- xmljson *
- alabaster ==0.7.12 develop
- appdirs ==1.4.4 develop
- arrow ==0.15.6 develop
- attrs ==19.3.0 develop
- babel ==2.8.0 develop
- binaryornot ==0.4.4 develop
- bump2version ==1.0.0 develop
- bumpversion ==0.6.0 develop
- certifi ==2020.4.5.1 develop
- chardet ==3.0.4 develop
- click ==7.1.2 develop
- cookiecutter ==1.7.2 develop
- coverage ==5.1 develop
- distlib ==0.3.0 develop
- docutils ==0.16 develop
- filelock ==3.0.12 develop
- idna ==2.9 develop
- imagesize ==1.2.0 develop
- importlib-metadata ==1.6.0 develop
- jinja2 ==2.11.2 develop
- jinja2-time ==0.2.0 develop
- markupsafe ==1.1.1 develop
- more-itertools ==8.2.0 develop
- packaging ==20.3 develop
- pluggy ==0.13.1 develop
- poyo ==0.5.0 develop
- py ==1.8.1 develop
- pygments ==2.6.1 develop
- pyparsing ==2.4.7 develop
- pytest ==5.4.2 develop
- python-dateutil ==2.8.1 develop
- python-slugify ==4.0.0 develop
- pytz ==2020.1 develop
- requests ==2.23.0 develop
- ruamel.yaml ==0.16.10 develop
- ruamel.yaml.clib ==0.2.0 develop
- six ==1.14.0 develop
- snowballstemmer ==2.0.0 develop
- sphinx ==3.0.3 develop
- sphinx-gallery ==0.6.2 develop
- sphinxcontrib-applehelp ==1.0.2 develop
- sphinxcontrib-devhelp ==1.0.2 develop
- sphinxcontrib-htmlhelp ==1.0.3 develop
- sphinxcontrib-jsmath ==1.0.1 develop
- sphinxcontrib-qthelp ==1.0.3 develop
- sphinxcontrib-serializinghtml ==1.1.4 develop
- text-unidecode ==1.3 develop
- toml ==0.10.1 develop
- tox ==3.15.0 develop
- urllib3 ==1.25.9 develop
- virtualenv ==20.0.20 develop
- wcwidth ==0.1.9 develop
- zipp ==3.1.0 develop
- argschema ==1.17.5
- certifi ==2020.4.5.1
- chardet ==3.0.4
- cycler ==0.10.0
- gitdb ==4.0.5
- gitpython ==3.1.2
- h5py ==2.10.0
- idna ==2.9
- joblib ==0.14.1
- kiwisolver ==1.2.0
- marshmallow ==2.19.2
- matplotlib ==3.2.1
- numpy ==1.18.4
- pandas ==1.0.3
- pyparsing ==2.4.7
- python-dateutil ==2.8.1
- pytz ==2020.1
- requests ==2.23.0
- scikit-learn ==0.23.0
- scipy ==1.4.1
- six ==1.14.0
- smmap ==3.0.4
- threadpoolctl ==2.0.0
- urllib3 ==1.25.9
- xarray ==0.15.1
- xmljson ==0.2.1
- GitPython *
- argschema *
- joblib *
- matplotlib *
- numpy *
- pandas *
- pillow *
- scikit-learn *
- scipy *
- xarray *
- xmljson *