Data

Tools

Indexes

Applications

Experiments

Open Source Science

timeeval

Evaluation Tool for Anomaly Detection Algorithms on Time Series

https://github.com/timeeval/timeeval

Science Score: 49.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 4 DOI reference(s) in README
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    1 of 10 committers (10.0%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (18.2%) to scientific vocabulary

Keywords

anomaly-detection benchmark-framework benchmarking dask dask-distributed distributed jupyter-notebooks numpy pandas python3 time-series time-series-analysis time-series-anomaly-detection

Keywords from Contributors

hack interpretability standardization meshing robust optim matrix-exponential bridges sequences log-parser
Last synced: 6 months ago · JSON representation

Repository

Evaluation Tool for Anomaly Detection Algorithms on Time Series

Basic Info
Statistics
  • Stars: 135
  • Watchers: 7
  • Forks: 18
  • Open Issues: 33
  • Releases: 0
Topics
anomaly-detection benchmark-framework benchmarking dask dask-distributed distributed jupyter-notebooks numpy pandas python3 time-series time-series-analysis time-series-anomaly-detection
Created about 4 years ago · Last pushed 6 months ago
Metadata Files
Readme License Citation

README.md

TimeEval logo

TimeEval

Evaluation Tool for Anomaly Detection Algorithms on Time Series.

[![CI](https://github.com/TimeEval/TimeEval/actions/workflows/build.yml/badge.svg)](https://github.com/TimeEval/TimeEval/actions/workflows/build.yml) [![Documentation Status](https://readthedocs.org/projects/timeeval/badge/?version=latest)](https://timeeval.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/TimeEval/TimeEval/branch/main/graph/badge.svg?token=esrQJQmMQe)](https://codecov.io/gh/TimeEval/TimeEval) [![!black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![PyPI version](https://badge.fury.io/py/TimeEval.svg)](https://badge.fury.io/py/TimeEval) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) ![python version 3.9|3.10|3.11|3.12](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue) [![Downloads](https://pepy.tech/badge/timeeval)](https://pepy.tech/project/timeeval)

See TimeEval Algorithms for algorithms that are compatible to this tool. The algorithms in that repository are containerized and can be executed using the DockerAdapter of TimeEval.

If you use TimeEval, please consider citing our paper.

TimeEval's documentation is hosted at https://timeeval.readthedocs.io.

Features

  • Large integrated benchmark dataset collection with more than 700 datasets
  • Benchmark dataset interface to select datasets easily
  • Adapter architecture for algorithm integration
    • DockerAdapter
    • JarAdapter
    • DistributedAdapter
    • MultivarAdapter
    • ... (add your own adapter)
  • Large collection of existing algorithm implementations (in TimeEval Algorithms repository)
  • Automatic algorithm detection quality scoring using AUC (Area under the ROC curve, also c-statistic) or range-based metrics
  • Automatic timing of the algorithm execution (differentiates pre-, main-, and post-processing)
  • Distributed experiment execution
  • Output and logfile tracking for subsequent inspection

Installation

TimeEval can be installed as a package or from source.

:warning: Attention!

Currently, TimeEval is tested only on Linux and macOS and relies on unixoid capabilities. On Windows, you can use TimeEval within WSL. If you want to use the provided detection algorithms, Docker is required.

Installation using pip

Builds of TimeEval are published to PyPI:

Prerequisites

  • python >= 3.9, < 3.13

:warning: Attention!

A dependency of TimeEval prevents us from supporting Python versions >= 3.13: prts is not updated and depends on NumPy<2.0.0. However, there is no NumPy version below 2.0.0 that supports Python>=3.13.

  • pip >= 20

  • Docker (for the anomaly detection algorithms)

  • (optional) rsync for distributed TimeEval

Steps

You can use pip to install TimeEval from PyPI:

sh pip install TimeEval

Installation from source

tl;dr

bash git clone git@github.com:TimeEval/TimeEval.git cd timeeval/ conda create -n timeeval python=3.9 conda activate timeeval pip install .

Prerequisites

The following tools are required to install TimeEval from source:

  • git
  • Python > 3.9 and Pip (anaconda or miniconda is preferred)

Steps

  1. Clone this repository using git and change into its root directory.

  2. Create a conda-environment and install all required dependencies:

sh conda create -n timeeval python=3.9 conda activate timeeval pip install .

  1. If you want to make changes to TimeEval or run the tests, you need to install the development dependencies with: pip install ".[ci]". The optional extra dependencies ".[dev]" contains additional dependencies for the notebooks and scripts packaged with TimeEval.

Usage

Example script:

```python from pathlib import Path from typing import Dict, Any

import numpy as np

from timeeval import TimeEval, DatasetManager, Algorithm, TrainingType, InputDimensionality from timeeval.adapters import FunctionAdapter from timeeval.algorithms import subsequence_if from timeeval.params import FixedParameters

Load dataset metadata

dm = DatasetManager(Path("tests/exampledata"), createif_missing=False)

Define algorithm

def myalgorithm(data: np.ndarray, args: Dict[str, Any]) -> np.ndarray: scorevalue = args.get("scorevalue", 0) return np.fulllike(data, fillvalue=scorevalue)

Select datasets and algorithms

datasets = dm.select() datasets = datasets[-1:]

Add algorithms to evaluate...

algorithms = [ Algorithm( name="MyAlgorithm", main=FunctionAdapter(myalgorithm), dataasfile=False, trainingtype=TrainingType.UNSUPERVISED, inputdimensionality=InputDimensionality.UNIVARIATE, paramconfig=FixedParameters({"scorevalue": 1.}) ), subsequenceif(params=FixedParameters({"n_trees": 50})) ] timeeval = TimeEval(dm, datasets, algorithms)

execute evaluation

timeeval.run()

retrieve results

print(timeeval.get_results()) ```

Citation

If you use TimeEval in your project or research, please cite our demonstration paper:

Phillip Wenig, Sebastian Schmidl, and Thorsten Papenbrock. TimeEval: A Benchmarking Toolkit for Time Series Anomaly Detection Algorithms. PVLDB, 15(12): 3678 - 3681, 2022. doi:10.14778/3554821.3554873

bibtex @article{WenigEtAl2022TimeEval, title = {TimeEval: {{A}} Benchmarking Toolkit for Time Series Anomaly Detection Algorithms}, author = {Wenig, Phillip and Schmidl, Sebastian and Papenbrock, Thorsten}, date = {2022}, journaltitle = {Proceedings of the {{VLDB Endowment}} ({{PVLDB}})}, volume = {15}, number = {12}, pages = {3678--3681}, doi = {10.14778/3554821.3554873} }

Owner

  • Name: TimeEval
  • Login: TimeEval
  • Kind: organization
  • Location: Germany

Time series anomaly detection tools from the HPI Information Systems group

GitHub Events

Total
  • Issues event: 4
  • Watch event: 29
  • Delete event: 14
  • Issue comment event: 15
  • Push event: 23
  • Pull request review event: 13
  • Pull request event: 27
  • Fork event: 3
  • Create event: 13
Last Year
  • Issues event: 4
  • Watch event: 29
  • Delete event: 14
  • Issue comment event: 15
  • Push event: 23
  • Pull request review event: 13
  • Pull request event: 27
  • Fork event: 3
  • Create event: 13

Committers

Last synced: over 1 year ago

All Time
  • Total Commits: 786
  • Total Committers: 10
  • Avg Commits per committer: 78.6
  • Development Distribution Score (DDS): 0.429
Past Year
  • Commits: 45
  • Committers: 6
  • Avg Commits per committer: 7.5
  • Development Distribution Score (DDS): 0.644
Top Committers
Name Email Commits
CodeLionX s****l@t****e 449
wenig i****o@p****e 147
CodeLionX s****l@h****e 98
Sebastian Schmidl C****X 44
dependabot[bot] 4****] 29
Phillip Wenig p****g@h****e 9
Shubham Mamgain 4****0 4
Phillip Wenig w****g 3
github-actions[bot] 4****] 2
Lukas Budach l****h@s****e 1
Committer Domains (Top 20 + Academic)
hpi.de: 2 student.hpi.uni-potsdam.de: 1 pwenig.de: 1 t-online.de: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 19
  • Total pull requests: 60
  • Average time to close issues: 5 days
  • Average time to close pull requests: 6 days
  • Total issue authors: 9
  • Total pull request authors: 7
  • Average comments per issue: 1.0
  • Average comments per pull request: 0.85
  • Merged pull requests: 56
  • Bot issues: 0
  • Bot pull requests: 36
Past Year
  • Issues: 3
  • Pull requests: 25
  • Average time to close issues: 6 days
  • Average time to close pull requests: 1 day
  • Issue authors: 3
  • Pull request authors: 4
  • Average comments per issue: 1.67
  • Average comments per pull request: 0.68
  • Merged pull requests: 23
  • Bot issues: 0
  • Bot pull requests: 21
View more stats
Top Authors
Issue Authors
  • CodeLionX (9)
  • SebastianSchmidl (2)
  • hadysysdev (2)
  • wenig (2)
  • hlj2546 (1)
  • gonzachiar (1)
  • zemogoravla (1)
  • fahmirzldi (1)
  • aleLoPrete (1)
Pull Request Authors
  • dependabot[bot] (45)
  • CodeLionX (21)
  • mshubham00 (11)
  • github-actions[bot] (8)
  • wenig (5)
  • SebastianSchmidl (3)
  • gonzachiar (2)
Top Labels
Issue Labels
:medal_sports: medium (6) comp: :book: documentation (5) comp: datasets (5) enhancement (4) help wanted (4) comp: algorithms (3) :medal_sports: low (2) :broken_heart: breaking change (2) comp: :hash: scoring (2) good first issue (2) :medal_sports: very low (1) question (1) :bug: bug(-fix) (1) comp: adapters (1) comp: TimeEval (general) (1) :medal_sports: high (1)
Pull Request Labels
dependencies (49) github_actions (22) comp: :book: documentation (13) comp: algorithms (11) enhancement (10) 🤖 bot (8) comp: :package: CI / project / packaging (7) :medal_sports: low (6) :bug: bug(-fix) (5) :broken_heart: breaking change (4) comp: :hash: scoring (4) comp: TimeEval (general) (4) comp: datasets (4) :medal_sports: high (4) :medal_sports: medium (3) comp: adapters (3) :medal_sports: very low (3)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 3,697 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 17
  • Total maintainers: 2
pypi.org: timeeval

Evaluation Tool for Time Series Anomaly Detection Methods

  • Versions: 17
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 3,697 Last month
Rankings
Stargazers count: 9.4%
Dependent packages count: 10.0%
Forks count: 13.3%
Average: 14.5%
Downloads: 17.9%
Dependent repos count: 21.8%
Maintainers (2)
CodeLionX wenig
Last synced: 6 months ago

Dependencies

.github/workflows/build.yml actions
  • RalfG/python-wheels-manylinux-build v0.4.2-manylinux2010_x86_64 composite
  • actions/checkout v3 composite
  • actions/download-artifact v3 composite
  • actions/upload-artifact v3 composite
  • codecov/codecov-action v3 composite
  • conda-incubator/setup-miniconda v2.2.0 composite
  • pypa/gh-action-pypi-publish v1.6.4 composite
  • schneegans/dynamic-badges-action v1.6.0 composite
.github/workflows/test-pr.yml actions
  • actions/checkout v3 composite
  • codecov/codecov-action v3 composite
  • conda-incubator/setup-miniconda v2.2.0 composite
docs/requirements.txt pypi
  • myst-parser ==0.18.0
  • sphinx ==5.0.2
  • sphinx_rtd_dark_mode ==1.2.4
  • sphinx_rtd_theme ==1.0.0
environment.yml pypi
  • click ==8.0.2
  • dask ==2021.5.1
  • docker >=4.4.0
  • durations >=0.3.0
  • numpyencoder >=0.3.0
  • prts ==1.0.0.3