automlbenchmark_early
Early stopping for the AMLB
Science Score: 54.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.5%) to scientific vocabulary
Repository
Early stopping for the AMLB
Basic Info
- Host: GitHub
- Owner: israel-cj
- License: mit
- Language: Python
- Default Branch: master
- Size: 112 MB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
docs/readme.md
AutoML Benchmark
The OpenML AutoML Benchmark provides a framework for evaluating and comparing open-source AutoML systems.
The system is extensible because you can add your own
AutoML frameworks and datasets. For a thorough explanation of the benchmark, and evaluation of results,
you can read our paper.
Automatic Machine Learning (AutoML) systems automatically build machine learning pipelines or neural architectures in a data-driven, objective, and automatic way. They automate a lot of drudge work in designing machine learning systems, so that better systems can be developed, faster. However, AutoML research is also slowed down by two factors:
We currently lack standardized, easily-accessible benchmarking suites of tasks (datasets) that are curated to reflect important problem domains, practical to use, and sufficiently challenging to support a rigorous analysis of performance results.
Subtle differences in the problem definition, such as the design of the hyperparameter search space or the way time budgets are defined, can drastically alter a task’s difficulty. This issue makes it difficult to reproduce published research and compare results from different papers.
This toolkit aims to address these problems by setting up standardized environments for in-depth experimentation with a wide range of AutoML systems.
Website: https://openml.github.io/automlbenchmark/index.html
Documentation: https://openml.github.io/automlbenchmark/docs/index.html
Installation: https://openml.github.io/automlbenchmark/docs/getting_started/
Features:
- Curated suites of benchmarking datasets from OpenML (regression, classification).
- Includes code to benchmark a number of popular AutoML systems on regression and classification tasks.
- New AutoML systems can be added
- Experiments can be run in Docker or Singularity containers
- Execute experiments locally or on AWS
Owner
- Name: israel-cj
- Login: israel-cj
- Kind: user
- Company: Eindhoven University of Technology
- Repositories: 10
- Profile: https://github.com/israel-cj
Citation (CITATION.cff)
cff-version: 1.2.0
message: "If you use this software, please cite it as below."
title: "AutoML Benchmark"
version: 2.1.7
license: "MIT"
url: "https://github.com/openml/automlbenchmark"
preferred-citation:
type: article
authors:
- family-names: "Gijsbers"
given-names: "Pieter"
orcid: "https://orcid.org/0000-0001-7346-8075"
- family-names: "de Paula Bueno"
given-names: "Marcos"
- family-names: "Coors"
given-names: "Stefan"
orcid: "https://orcid.org/0000-0001-7346-8075"
- family-names: "LeDell"
given-names: "Erin"
- family-names: "Poirier"
given-names: "Sébastien"
- family-names: "Thomas"
given-names: "Janek"
orcid: "https://orcid.org/0000-0003-4511-6245"
- family-names: "Bischl"
given-names: "Bernd"
orcid: "https://orcid.org/0000-0001-6002-6980"
- family-names: "Vanschoren"
given-names: "Joaquin"
orcid: "https://orcid.org/0000-0001-7044-9805"
journal: "Journal of Machine Learning Research"
start: 1 # First page number
end: 65 # Last page number
title: "AMLB: an AutoML Benchmark"
issue: 101
volume: 25
year: 2024
url: http://jmlr.org/papers/v25/22-0493.html