automlbenchmark_normal
Science Score: 54.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.5%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: israel-cj
- License: mit
- Language: Python
- Default Branch: master
- Size: 112 MB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
docs/readme.md
AutoML Benchmark
The OpenML AutoML Benchmark provides a framework for evaluating and comparing open-source AutoML systems.
The system is extensible because you can add your own
AutoML frameworks and datasets. For a thorough explanation of the benchmark, and evaluation of results,
you can read our paper.
Automatic Machine Learning (AutoML) systems automatically build machine learning pipelines or neural architectures in a data-driven, objective, and automatic way. They automate a lot of drudge work in designing machine learning systems, so that better systems can be developed, faster. However, AutoML research is also slowed down by two factors:
We currently lack standardized, easily-accessible benchmarking suites of tasks (datasets) that are curated to reflect important problem domains, practical to use, and sufficiently challenging to support a rigorous analysis of performance results.
Subtle differences in the problem definition, such as the design of the hyperparameter search space or the way time budgets are defined, can drastically alter a task’s difficulty. This issue makes it difficult to reproduce published research and compare results from different papers.
This toolkit aims to address these problems by setting up standardized environments for in-depth experimentation with a wide range of AutoML systems.
Website: https://openml.github.io/automlbenchmark/index.html
Documentation: https://openml.github.io/automlbenchmark/docs/index.html
Installation: https://openml.github.io/automlbenchmark/docs/getting_started/
Features:
- Curated suites of benchmarking datasets from OpenML (regression, classification).
- Includes code to benchmark a number of popular AutoML systems on regression and classification tasks.
- New AutoML systems can be added
- Experiments can be run in Docker or Singularity containers
- Execute experiments locally or on AWS
Owner
- Name: israel-cj
- Login: israel-cj
- Kind: user
- Company: Eindhoven University of Technology
- Repositories: 10
- Profile: https://github.com/israel-cj
Citation (CITATION.cff)
cff-version: 1.2.0
message: "If you use this software, please cite it as below."
title: "AutoML Benchmark"
version: 2.1.7
license: "MIT"
url: "https://github.com/openml/automlbenchmark"
preferred-citation:
type: article
authors:
- family-names: "Gijsbers"
given-names: "Pieter"
orcid: "https://orcid.org/0000-0001-7346-8075"
- family-names: "de Paula Bueno"
given-names: "Marcos"
- family-names: "Coors"
given-names: "Stefan"
orcid: "https://orcid.org/0000-0001-7346-8075"
- family-names: "LeDell"
given-names: "Erin"
- family-names: "Poirier"
given-names: "Sébastien"
- family-names: "Thomas"
given-names: "Janek"
orcid: "https://orcid.org/0000-0003-4511-6245"
- family-names: "Bischl"
given-names: "Bernd"
orcid: "https://orcid.org/0000-0001-6002-6980"
- family-names: "Vanschoren"
given-names: "Joaquin"
orcid: "https://orcid.org/0000-0001-7044-9805"
journal: "Journal of Machine Learning Research"
start: 1 # First page number
end: 65 # Last page number
title: "AMLB: an AutoML Benchmark"
issue: 101
volume: 25
year: 2024
url: http://jmlr.org/papers/v25/22-0493.html
GitHub Events
Total
Last Year
Dependencies
- actions/checkout v3 composite
- actions/configure-pages v3 composite
- actions/deploy-pages v2 composite
- actions/setup-python v4 composite
- actions/upload-pages-artifact v2 composite
- actions/cache v3 composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- actions/cache v3 composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- actions/checkout v3 composite
- actions/checkout v3 composite
- actions/github-script v6 composite
- author/action-rollback stable composite
- scikit-learn ==1.3.1
- packaging *
- scikit-learn ==1.1.0
- colorama >=0.3.8
- future *
- packaging *
- pandas *
- requests >=2.10
- tabulate *
- liac-arff ==2.4
- numpy >=1.15,<2.0
- pandas >=0.23,<1.0
- ruamel.yaml >=0.15,<1.0
- scikit-learn >=0.22.2
- scipy >=1.5,<1.6
- setuptools *
- torch >=1.6.0,<1.7.0
- tpot >=0.11.0,<0.12
- xgboost >=1.1.0,<1.2
- lccv *
- pandas *
- pandas *
- openml *
- packaging <22.0
- rpy2 ==2.3.0
- rpy2 ==2.3.0
- psutil >=5.4
- pyarrow >=4.0
- numpy ==1.24.2
- psutil ==5.8.0
- pyarrow ==11.0.0
- boto3-stubs * development
- mkdocs-material * development
- mypy * development
- pandas-stubs * development
- pip-tools * development
- pytest * development
- pytest-mock * development
- types-psutil * development
- types-xmltodict * development
- matplotlib *
- numpy *
- openml *
- pandas *
- seaborn *
- tabulate *
- boto3 >=1.9,<2.0
- filelock >=3.4
- fsspec *
- liac-arff >=2.5,<3.0
- numpy >=1.24,<2.0
- openml ==0.13.1
- pandas >=1.5,<2.0
- psutil >=5.4,<6.0
- pyarrow >=11.0
- ruamel.yaml >=0.15,<1.0
- s3fs *
- scikit-learn >=1.0,<2.0
- boto3 ==1.26.98
- botocore ==1.29.98
- certifi ==2022.12.7
- charset-normalizer ==3.1.0
- filelock ==3.12.0
- fsspec ==2023.6.0
- idna ==3.4
- jmespath ==1.0.1
- joblib ==1.2.0
- liac-arff ==2.5.0
- minio ==7.1.13
- numpy ==1.24.2
- openml ==0.13.1
- pandas ==1.5.3
- psutil ==5.9.4
- pyarrow ==11.0.0
- python-dateutil ==2.8.2
- pytz ==2022.7.1
- requests ==2.28.2
- ruamel-yaml ==0.17.21
- ruamel-yaml-clib ==0.2.7
- s3fs ==0.4.2
- s3transfer ==0.6.0
- scikit-learn ==1.2.2
- scipy ==1.10.1
- six ==1.16.0
- threadpoolctl ==3.1.0
- urllib3 ==1.26.15
- xmltodict ==0.13.0