Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of DeepHyper

DeepHyper - 0.11.0

What's Changed

  • feat (search): solution selection columns are not performed by default by @Deathn0t in https://github.com/deephyper/deephyper/pull/335
  • fix (cbo): fitsurrogate can now process dataframe with only failures as input, refactoring maxtotal_failures by @Deathn0t in https://github.com/deephyper/deephyper/pull/337
  • test (callback): add test for CSVLoggerCallback by @bretteiffert in https://github.com/deephyper/deephyper/pull/344
  • chore (logging): replace context.yaml with logging by @wigging in https://github.com/deephyper/deephyper/pull/343
  • fix (tests): use tmp_path for params test by @wigging in https://github.com/deephyper/deephyper/pull/345
  • fix (numpyro): temporary workaround with numpyro and jax 0.7.0 by @Deathn0t in https://github.com/deephyper/deephyper/pull/347
  • docs (examples): add example for hpo with stopper by @bretteiffert in https://github.com/deephyper/deephyper/pull/346
  • refactor (logger): use module level logger instead of root level by @wigging in https://github.com/deephyper/deephyper/pull/349
  • chore (docs): mirror content from test_quickstart.py to index.rst by @bretteiffert in https://github.com/deephyper/deephyper/pull/350
  • feat (search): decouple search and ask/tell methods by @Deathn0t in https://github.com/deephyper/deephyper/pull/353

Full Changelog: https://github.com/deephyper/deephyper/compare/0.10.1...0.11.0

Breaking changes

DeepHyper now accepts two interfaces for the search. Instead of giving the evaluator as an input of the Search(..., evaluator) (DEPRECATED!) you now need to pass the evaluator as an input of the search.search(evaluator, max_evals) method.

Ask and Tell Interface (NEW)

The first interface is the classic configurations = search.ask(...) and search.tell(configurations_with_objective). In this case, you need to manage the computation of objectives yourself. This interface is more flexible. However, asynchronous parallel evaluations are not managed for you. You also need to be careful on the behaviour of the ask/tell methods that are algorithm's dependent. For example, ask can return the same value if called multiple times until tell is called (in Bayesian optimization, it updates the internal surrogate model state).

Here is an example:

python search = create_search() max_evals = 100 for i in range(max_evals): # The output is a list of dict config = search.ask(1)[0] y = -config["x"] ** 2 # The input is a list of tuple with dict and objective(s) print(f"[{i=:03d}] >>> f(x={config['x']:.3f}) = {y:.3f}") search.tell([(config, y)])

Search Interface

The second interface is results = search.search(evaluator, ...) that is bonded to the evaluator and manages the loop for asynchronous parallel evaluations for you. We can execute the search for a given number of iterations by using the search.search(evaluator, max_evals=...). It is also possible to use the timeout parameter if one needs a specific time budget (e.g., restricted computational time in machine learning competitions, allocation time in HPC).

python search = create_search() results = search.search(evaluator, max_evals)

Scientific Software - Peer-reviewed - Python
Published by Deathn0t 4 months ago

DeepHyper - 0.10.2

What's Changed

  • feat (CBO): adding outliersiqrfactor to CBO by @Deathn0t in https://github.com/deephyper/deephyper/pull/324
  • tutorial: updated queued evaluator mpi tutorial by @bretteiffert in https://github.com/deephyper/deephyper/pull/327
  • feat (search): adding option to disable dumping the results to civ by @wigging in https://github.com/deephyper/deephyper/pull/325
  • feat (search): adding solution selection through model for single objective optimization by @Deathn0t in https://github.com/deephyper/deephyper/pull/329
  • chore (doc): updating quickstart in landing page and readme with new solution selection in results by @Deathn0t in https://github.com/deephyper/deephyper/pull/331

Full Changelog: https://github.com/deephyper/deephyper/compare/0.10.1...0.10.2

Scientific Software - Peer-reviewed - Python
Published by Deathn0t 6 months ago

DeepHyper - 0.10.1

What's Changed]

  • Remove tensorflow installs and references by @wigging in https://github.com/deephyper/deephyper/pull/292
  • Fix example on notification of failures in hpo by @Deathn0t in https://github.com/deephyper/deephyper/pull/293
  • Fixes several issues in the documentation build by @Deathn0t in https://github.com/deephyper/deephyper/pull/294
  • Fix documentation issues and cleaning code by @Deathn0t in https://github.com/deephyper/deephyper/pull/295
  • convert and move hpo text classification tutorial to example in new format by @bretteiffert in https://github.com/deephyper/deephyper/pull/298
  • Refactoring parameters of CBO and deephyper.skopt.optimizer by @Deathn0t in https://github.com/deephyper/deephyper/pull/299
  • Fix shared memory for multiple node comm with MPIWinStorage by @Deathn0t in https://github.com/deephyper/deephyper/pull/301
  • Optimization for RMA window memory allocation by @Deathn0t in https://github.com/deephyper/deephyper/pull/306
  • Fix ValueError in example by @Deathn0t in https://github.com/deephyper/deephyper/pull/307
  • Improve hpo text classification tutorial by @bretteiffert in https://github.com/deephyper/deephyper/pull/302
  • Joss tutorial cleanup by @bretteiffert in https://github.com/deephyper/deephyper/pull/316
  • Integration of JOSS submition by @Deathn0t in https://github.com/deephyper/deephyper/pull/322

Full Changelog: https://github.com/deephyper/deephyper/compare/0.9.3...0.10.1

Scientific Software - Peer-reviewed - Python
Published by Deathn0t 8 months ago

DeepHyper - 0.10.0

What's Changed

  • Remove tensorflow installs and references by @wigging in https://github.com/deephyper/deephyper/pull/292
  • Fix example on notification of failures in hpo by @Deathn0t in https://github.com/deephyper/deephyper/pull/293
  • Fixes several issues in the documentation built by @Deathn0t in https://github.com/deephyper/deephyper/pull/294
  • Fix documentation issues and clean code by @Deathn0t in https://github.com/deephyper/deephyper/pull/295
  • Convert and move the HPO text classification tutorial to example in new format by @bretteiffert in https://github.com/deephyper/deephyper/pull/298
  • Refactoring parameters of CBO and deephyper.skopt.optimizer by @Deathn0t in https://github.com/deephyper/deephyper/pull/299

Full Changelog: https://github.com/deephyper/deephyper/compare/0.9.3...0.10.0

Scientific Software - Peer-reviewed - Python
Published by Deathn0t 9 months ago

DeepHyper - 0.9.3

What's Changed

  • Simplify decentralized search by @Deathn0t in https://github.com/deephyper/deephyper/pull/260
  • Add uv installation page in documentation by @wigging in https://github.com/deephyper/deephyper/pull/263
  • Changes for Colab tutorial by @bretteiffert in https://github.com/deephyper/deephyper/pull/271
  • Add section in documentation for facility guides by @wigging in https://github.com/deephyper/deephyper/pull/265, https://github.com/deephyper/deephyper/pull/276
  • Add parameters_at_max function for Quick Start example by @wigging in https://github.com/deephyper/deephyper/pull/275
  • Fix ensemble inference by @Deathn0t in https://github.com/deephyper/deephyper/pull/278
  • Move BBO tutorial to Sphinx Gallery by @Deathn0t in https://github.com/deephyper/deephyper/pull/280
  • Adding LokyEvaluator by @Deathn0t in https://github.com/deephyper/deephyper/pull/281
  • Add Contrib documentation about gallery examples by @Deathn0t in https://github.com/deephyper/deephyper/pull/282
  • Move ensemble tutorials to Sphinx Gallery by @Deathn0t in https://github.com/deephyper/deephyper/pull/283
  • Removing tutorials and dependices tf-keras2 by @Deathn0t in https://github.com/deephyper/deephyper/pull/284
  • Replacing sdv by Gaussian Mixture Model and removing dependencies by @Deathn0t in https://github.com/deephyper/deephyper/pull/286
  • Move multi-objective BBBO tutorial to Sphinx Gallery by @wigging in https://github.com/deephyper/deephyper/pull/285
  • Update example on failures in BBO by @Deathn0t in https://github.com/deephyper/deephyper/pull/287
  • Move overfitting tutorial to Sphinx Gallery by @Deathn0t in https://github.com/deephyper/deephyper/pull/289

Full Changelog: https://github.com/deephyper/deephyper/compare/0.9.2...0.9.3

Scientific Software - Peer-reviewed - Python
Published by Deathn0t 10 months ago

DeepHyper - Changelog - DeepHyper 0.9.2

Scientific Software - Peer-reviewed - Python
Published by Deathn0t 10 months ago

DeepHyper - Changelog - DeepHyper 0.9.1

What's Changed

  • Remove tox file, unittest, and revise workflow by @wigging in https://github.com/deephyper/deephyper/pull/247
  • Add tests workflow for macOS by @wigging in https://github.com/deephyper/deephyper/pull/254
  • Add tests workflow for MPI by @Deathn0t in https://github.com/deephyper/deephyper/pull/249
  • Add tests workflow for Redis testing by @Deathn0t in https://github.com/deephyper/deephyper/pull/253
  • Add tests workflow for core extras by @Deathn0t in https://github.com/deephyper/deephyper/pull/256
  • Add support for ForbiddenClause When Using Mixedga/ga optimisers in CBO by @Deathn0t in https://github.com/deephyper/deephyper/pull/257
  • Add example for scaling centralized Bayesian optimization
  • Add example for scaling Bayesian optimization with decentralized search by @Deathn0t in https://github.com/deephyper/deephyper/pull/258
  • Resolved bug with termination condition max_evals in the case of decentralized search by @Deathn0t https://github.com/deephyper/deephyper/pull/259
  • Updated spack package recipies by @bretteiffert in https://github.com/deephyper/deephyper-spack-packages/pull/4

Full Changelog: https://github.com/deephyper/deephyper/compare/0.9.0...0.9.1

Scientific Software - Peer-reviewed - Python
Published by Deathn0t 12 months ago

DeepHyper - Changelog - DeepHyper 0.9.0

We are happy to release the new version of DeepHyper with software quality updates.

  • DeepHyper is now compatible with Python>=3.10;<=3.13.
  • The pip installation was updated.
  • The package build tools have been updated to modern tools (mainly, pyproject.toml, hatchling, and ruff). @wigging
  • The code base style/content was improved accordingly. @wigging
  • Our contributing guidelines have been updated. @wigging
  • The CI tests pipeline has been updated. @wigging

deephyper.evaluator

The Evaluator API has been updated to avoid possibly leaking threads when using search.search(timeout=...). The "serial" method is now only accepting coroutinefunction (i.e., using the async def keywords in the function definition). The running job received by the run-function def run(job) now has job.status (READY, RUNNING, DONE, CANCELLING, CANCELLED) to handle coopérative cancellation. The user can regularly check this status to manage the job termination (useful with parallel backends such as "serial", "thread", "mpicomm". The dumped results.csv will now provide the final status of each job. @Deathn0t

New examples will be provided in the documentation.

deephyper.evaluator.storage

Two new Storage are now available to benefit from shared memory in distributed parallel execution. - MPIWinStorage: specific to "mpicomm" execution and based on one-sided communication (a.k.a., remote memory access RMA). @Deathn0t - SharedMemoryStorage: specific to "process" execution. @Deathn0t

Removing deprecated modules

The deephyper.search and deephyper.problem subpackages previously deprecated are now removed. The deephyper.hpo should be used instead.

Spack installation

Our team is currently undergoing an update of our Spack installation. If you are a Spack user and would like to experiment with it, please get in touch with us. @bretteiffert

Scientific Software - Peer-reviewed - Python
Published by Deathn0t about 1 year ago

DeepHyper - Changelog - DeepHyper 0.8.1

We are happy to release the new version of DeepHyper with light updates bug-fixes.

  • fixing issues with constant hyperparameters using conditions c761d3f63c91a069370fb163f6a65e23d6228c2a
  • Ensemble can use PredictorLoader for lazy loading 9755cd09bb87904674d7b25ef9eeb42793ee322f
  • fixing issue in SklearnPredictor 788e23827d24f7a6d61606f0a8189ec5ad0b05d8
  • making MeanAggregator compatible with MaskedArray for predictions with variable targets (e.g., cross-validation, boostrap) e3b4408b6857ff8b2a714ea2671baaa4efc00aca
  • adding OnlineSelector to combine online ensemble selection with hyperparameter optimization 4ba8d11f2604a905001ab7eb09c95d0991f8ae67
  • Evaluator can now be used to schedule any parallel function call and be used outside HPO (for example it is now use within Ensemble to schedule parallel predictions of members).
  • adding more regularization hyperparameters to greedy ensemble selection 77b6e5049fdb50fd0b5cdfa70d33dd28b690e760

Scientific Software - Peer-reviewed - Python
Published by Deathn0t about 1 year ago

DeepHyper - Changelog - DeepHyper 0.8.0

We are happy to release the new version of DeepHyper with significant updates that improve the consistency of our package and make it interoperable with more machine-learning frameworks.

The most significant updates are for deephyper.hpo and deephyper.ensemble.

We have included for you below the details about the update.

deephyper.analysis

  • deephyper.analysis.hpo (link to documentation)includes new utility functions:
    • filter_failed_objectives
    • parameters_at_max
    • parameters_at_topk
    • parameters_from_row
    • read_results_from_csv

deephyper.ensemble

The ensemble subpackage (link to documentation) was refactored to expand its applicability. It is now compatible with any ML framework such as Tensorflow/Keras2, PyTorch, and Scikit-Learn models. The only requirement is to follow the new deephyper.predictor interface that represents a frozen machine learning model on which inferences can be run.

To help you start with the new ensemble feature we prepared a tutorial that optimizes the hyperparameter of a Decision Tree and build an ensemble to improve accuracy, probabilistic calibration, and uncertainty estimates: Hyperparameter Optimization for Tree Ensemble with Uncertainty Quantification (Scikit-Learn)

The ensemble API is mainly built around the follow classes: - Predictor that represents a predictive function (see doc on predictor). - Aggregator: to aggregate the predictions from a set of predictors (see doc on aggregator). - Loss: loss and scoring functions adapted to classification, regression, or uncertainty quantification (see doc on loss). - Selector: a selection algorithm that selects a subset of predictors (see doc on selector) and weight them.

The API already provides the tools to build: 1. Simple ensembles to improve the predictions of models with variability (when retrained). Our tutorial on Decision trees is a good example of that. 2. Ensembles with Epistemic uncertainty. 3. Ensembles with decomposed aleatoric and epistemic uncertainty.

For classification, you can use the MixedCategoricalAggregator (see doc) that can use confidence or entropy for uncertainty.

For regression, you can use the MixedNormalAggregator (see doc) that uses variance for uncertainty.

deephyper.hpo

The deephyper.hpo comes in replacement of both deephyper.search.hps and deephyper.search.nas. For consistency, we decided to refactor neural architecture search and hyperparameter optimization together. The main algorithms to explore hyperparameters are:

All search algorithms are now following the _ask/_tell interface. Good and simple examples to follow if you want to implement your own algorithm are RandomSearch and RegularizedEvolution).

Therefore all search algorithms are now compatible with decentralization. An example for decentralized search with subprocess is available here.

A new tutorial on how to do neural architecture search was published: Neural Architecture Search with Tensorflow/Keras2 (Basic).

deephyper.predictor

Utility classes are provided to reload checkpointed models from Scikit-Learn, Tensorflow/Keras2 and Pytorch.

deephyper.stopper

This sub-package provides algorithm that can help speed-up hyperparameter optimization by observing the training of machine learning models.

The main update is to expose the BayesianLearningCurveRegressor which helps extrapolate with uncertainty the future performance of a training curve based on parametric models.

The currently available algorithms are: - Learning Curve Extrapolation (see doc on LCModelStopper). - Median Stopper (see doc on MedicanStopper). - Asynchronous Successive Halving (see doc on SuccessiveHalvingStopper).

Scientific Software - Peer-reviewed - Python
Published by Deathn0t over 1 year ago

DeepHyper - Changelog - DeepHyper 0.7.0

deephyper.search

  • If a results.csv file already exists in the log_dir folder, it is renamed instead of overwritten.
  • Parallel and asynchronous standard experimental design can now be used through DeepHyper to perform Random, Grid, or Quasi-Monte-Carlo evaluations: Example on Standard Experimental Design (Grid Search).

Bayesian optimization (CBO and MPIDistributedBO)

  • New optimizers of the acquisition function: the acquisition function for non-derivable surrogate models (e.g., "RF", "ET") can now be optimized with acq_optimizer="ga" or acq_optimizer="mixedga". This makes BO iterations more efficient but implies an additional overhead (negligible if the evaluated function is slow). The acq_optimizer_freq=2 parameter can be used to amortize this overhead.
  • New exploration/exploitation scheduler: the periodic exponential decay scheduler can now be specified with its initial and final values CBO(..., kappa=10, scheduler={"type": "periodic-exp-decay", "period": 25, "kappa_final": 1.96}). This mechanism allows to escape local optimum.
  • New family of acquisition functions for Random-Forests surrogate models. acq_func="UCBd" or "EId" or "PId". The "d" postfix stands for "deterministic" because in this case, the acquisition function will only use epistemic uncertainty from the black-box function to evaluate the acquisition function and it will ignore aleatoric uncertainty (i.e., noise estimation).
  • The default surrogate model was renamed "ET" which stands for "Extremely Randomized Trees" to better match the machine learning literature. This surrogate model provides better epistemic uncertainty estimates than the standard "RF" which stands for "Random Forest". It is also a type of randomized ensemble of trees but it uses a randomized split decision rule instead of an optimized split decision rule.
  • HpProblem based on ConfigSpace objects using constraints now uses the lower bound of each hyperparameter as a slack value.

deephyper.stopper

  • The stopper based on learning curve extrapolation has an improved fit and speed.

Scientific Software - Peer-reviewed - Python
Published by Deathn0t over 1 year ago

DeepHyper - Changelog - DeepHyper 0.6.0

  • The documentation site theme was updated.
  • PyPI Release: https://pypi.org/project/deephyper/0.6.0/
  • New BibTeX citation for DeepHyper to include our growing community:

@misc{deephyper_software, title = {"DeepHyper: A Python Package for Scalable Neural Architecture and Hyperparameter Search"}, author = {{DeepHyper Development Team}}, organization = {DeepHyper Team}, year = 2018, url = {https://github.com/deephyper/deephyper} }

deephyper.evaluator

  • @profile(memory=True) decorator can now profile memory using tracemalloc (adding an overhead).
  • RayStorage is now available for the ray parallel backend. It is based on remote actors and is a wrapper around the base MemoryStorage. This allows to use deephyper.stopper in parallel only with ray backend requirements.

deephyper.search

  • Multi-objective optimization (MOO) has been upgraded for better optimization performance. A new tutorial to discover this feature is available at Multi-Objective Optimization - 101.
    • A minimum-lower bound performance can be specified to avoid exploring not interesting trade-offs moo_lower_bounds=....
    • A new objective scaler is available to normalize objectives (e.g., accuracy and latency) more efficiently objective_scaler="quantile-uniform".
    • The results.csv or DataFrame now contains a new information pareto_efficient which indicates the optimal solution in a multi-objective problem (i.e., Pareto-set/front).
  • Random-Forest (RF) surrogate model predictions are faster by about x1.5 factor, speeding up the Bayesian optimization process.

- Added a dynamic prior update for Bayesian optimization: update_prior=..., update_prior_quantile=... This allows to increase the density of sampling in areas of interest and makes "random"-sampling-based optimization of the surrogate model more competitive (against more expensive optimizers like gradient-based or genetic algorithms).

deephyper.stopper

  • SuccessiveHalvingStopper is now compatible with failures. If a "failure" is observed during training (i.e., observation starting with "F") then previous observations are replaced in shared memory to notify other competitors of the failure.

deephyper.analysis

  • Creation of a new module to provide utilities for the analysis of experiments.

Scientific Software - Peer-reviewed - Python
Published by Deathn0t over 2 years ago

DeepHyper - Changelog - DeepHyper 0.5.0

deephyper.evaluator

  • removed SubprocessEvaluator evaluator due to limited features and confusion with ProcessPoolEvaluator. The ProcessPoolEvaluator seamed to be enough for our different use cases (hyperparameter optimization, auto-tuning).
  • retrieving and returning remote exception when using MPICommEvaluator.
  • timeout of the search is now handled through thread-based timeout instead of signal handlers.
  • timeout with MPICommEvaluator is now handled through refreshed "remaining time" and thread-based timeout in each rank.
  • exceptions happening in remote rank when using MPICommEvaluator are now traced-back and printed out in root rank.
  • possibility to return metadata logged in the results DataFrame with the run-function.

python def run(job): config = job.parameters return {"objective": config["x"], "metadata": {"time": time.time()}} - new in the resutls.csv or returned pd.DataFrame, the hyperparameters will start with p: prefix and metadata will start with m: to allow for easier filtration of the columns. - new deephyper.evaluator.storage. The interface is defined by Storage with two basic implementations MemoryStorage (local memory) and RedisStorage (in-memory key-value database). - new RunningJob API. The run-function is now passed a RunningJob instance instead of dict. The RunningJob.parameters corresponds to the former dictionary passed to the run-function. The RunningJob object should implement the dictionary interface to be backward compatible with the previous standard argument of the run-function.

deephyper.search

  • new preprocessing of the objective values when using surrogate="RF" in CBO. The objective is now preprocessed with Min-Max and Log to improve the fitting of the surrogate model on small values and improve Bayesian optimisation convergence. (see paper)
  • new cyclic exponential decay of the exploration-exploitation trade-off in CBO this is particularly when using MPIDistributedDBO and scaling the number of BO instances to avoid "over-exploration".
  • new distributed Bayesian optimization through MPI and Storage available. See Tutorial - Distributed Bayesian Optimization with MPI and Redis.

deephyper.stopper

A new module in DeepHyper to allow for Multi-Fidelity Bayesian Optimization. A stopper can observe the evolving performance of an iterative algorithm and decide to continue its evaluation or stop it early. This can allow to the search to be more effective when the time-ressource or computational ressource is a bottleneck. However, it can also converge to sub-optimal solution. Different, multi-fidelity schemes are now proposed and documented at DeepHyper Documentation - Stopper.

Scientific Software - Peer-reviewed - Python
Published by Deathn0t almost 3 years ago

DeepHyper - Changelog - DeepHyper 0.4.2

deephyper.evaluator

  • patched ThreadPoolEvaluator to remove extra overheads of pool initialisation

deephyper.search

  • resolved constant hyperparameter when hyperparameter is discrete with log-uniform in df01040d44a8f3b80700f2f853a6b452680e1112
  • patch id to job_id in neural architecture search history saver
  • adding multi objectives optimisation to CBO a run-function can now return multiple objectives as a tuple to be maximised

python def run(config): ... return objective_0, objective_1

deephyper.ensemble

  • ensemble with uncertainty quantification UQBaggingEnsembleRegressor is now compatible with predictions of arbitrary shapes

deephyper dashboard

  • adding dashboard with deephyper-analytics dashboard paired with results stored in local deephyper database managed through DBManager
  • adding dataframe visualization
  • adding scatter plot visualization

Scientific Software - Peer-reviewed - Python
Published by Deathn0t over 3 years ago

DeepHyper - Changelog - DeepHyper 0.4.0

global updates

  • contributors of DeepHyper now appear on a dedicated page, see DeepHyper Authors, submit a PR if we forgot you!
  • lighter installation via pip install deephyper packed with the minimum requirements for hyperparameter search.
  • update API documentation
  • removed deephyper.benchmark
  • make neural architecture search features optional with pip install deephyper[nas]
  • make auto-sklearn features optional with pip install deephyper[popt] (Pipeline OPTimization)
  • improve epistemic uncertainty quantification for Random Forest surrogate model in Bayesian Optimisation
  • moved deephyper/scikit-optimize as a sub package deephyper.skopt
  • new tutorials dedicated to ALCF systems, see Tutorials - Argonne Leadership Computing Facility

deephyper.search

  • renamed AMBS to CBO (Centralised Bayesian Optimization) at deephyper.search.hps.CBO
  • added new scalable Distributed Bayesian Optimization algorithm at deephyper.search.hps.DBO (experimented with up to 4,096 parallel workers)
  • moved problem.add_starting_point of HpProblem to CBO(..., initial_points=[...])
  • added generative-model based transfer-learning for hyper-parameter optimisation (Example - Transfer Learning for Hyperparameter Search)
  • added filtration of duplicated configurations in CBO CBO(..., filter_duplicated=True)
  • notify failures to the optimiser for it to learn them (Example - Notify Failures in Hyperparameter optimization)
  • added new multi-point acquisition strategy for better scalability in CBO CBO(..., acq_func="UCB, multi_point_strategy="qUCB", ...)
  • added the possibility to switch between synchronous/asynchronous communication in CBO CBO(..., sync_communication=True, ...)

deephyper.evaluator

  • added MPI-based Evaluators (better scaling, lower initialisation overhead): MPICommEvaluator and MPIPoolEvaluator
  • added @profile(run_function) decorator for run-function to collect execution times/durations of the black-box, this allow to profile the worker utilisation (Example - Profile the Worker Utilisation)
  • added @queued(Evaluator) decorator for any evaluator class to manage a queue of resources
  • added SerialEvalutor to adapt to serial-search (one by one)
  • added deephyper.evaluator.callback.TqdmCallback to display progress bar when running a search
  • the run-function can now return other values than the objective to be logged in the results.csv for example {"objective": 0.9, "num_parameters": 20000, ...}
  • asyncio is patched automatically when using notebooks/ipython

deephyper.ensemble

Scientific Software - Peer-reviewed - Python
Published by Deathn0t over 3 years ago

DeepHyper - Changelog - DeepHyper 0.3.3

  • Now compatible with Python >=3.7, <3.10
  • Fixed log_dir argument in search
  • Added logging for command line HPS/NAS

Scientific Software - Peer-reviewed - Python
Published by Deathn0t about 4 years ago

DeepHyper - Changelog - DeepHyper 0.3.2

  • All the search algorithms were tested to have a correct behaviour when random_state is set.
  • Callbacks (deephyper.evaluator.callback) can now be used to extend the behavior of the existing Evaluator. A LoggerCallback, ProfilingCallback, SearchEarlyStopping are already available (see example below).
  • All search algorithms are now importable from their hps or nas package. For example, from deephyper.search.hps import AMBS and from deephyper.search.nas import AgEBO.
  • HpProblem and NaProblem do not have a seed parameter anymore. The random_state has to be set when instantiating a Search(random_state=...).

Examlpe: SearchEarlyStopping

```python from deephyper.problem import HpProblem from deephyper.search.hps import AMBS from deephyper.evaluator import Evaluator from deephyper.evaluator.callback import LoggerCallback, SearchEarlyStopping

problem = HpProblem() problem.add_hyperparameter((0.0, 10.0), "x")

def f(config): return config["x"]

evaluator = Evaluator.create(f, method="ray", methodkwargs={ "numcpus": 1, "numcpuspertask": 0.25, "callbacks": [LoggerCallback(), SearchEarlyStopping(patience=10)] }) print(f"Num. Workers {evaluator.numworkers}")

search = AMBS(problem, evaluator, filter_duplicated=False)

results = search.search(max_evals=500) ```

Gives the following output:

console Num. Workers 4 [00001] -- best objective: 3.74540 -- received objective: 3.74540 [00002] -- best objective: 6.38145 -- received objective: 6.38145 Objective has improved from 3.74540 -> 6.38145 [00003] -- best objective: 6.38145 -- received objective: 3.73641 [00004] -- best objective: 7.29998 -- received objective: 7.29998 Objective has improved from 6.38145 -> 7.29998 [00005] -- best objective: 7.29998 -- received objective: 2.98912 [00006] -- best objective: 7.29998 -- received objective: 5.52077 [00007] -- best objective: 7.29998 -- received objective: 4.59535 [00008] -- best objective: 7.29998 -- received objective: 5.28775 [00009] -- best objective: 7.29998 -- received objective: 5.52099 [00010] -- best objective: 9.76781 -- received objective: 9.76781 Objective has improved from 7.29998 -> 9.76781 [00011] -- best objective: 9.76781 -- received objective: 7.48943 [00012] -- best objective: 9.76781 -- received objective: 7.42981 [00013] -- best objective: 9.76781 -- received objective: 9.30103 [00014] -- best objective: 9.76781 -- received objective: 8.22588 [00015] -- best objective: 9.76781 -- received objective: 8.96084 [00016] -- best objective: 9.76781 -- received objective: 8.96303 [00017] -- best objective: 9.96415 -- received objective: 9.96415 Objective has improved from 9.76781 -> 9.96415 [00018] -- best objective: 9.96415 -- received objective: 9.58723 [00019] -- best objective: 9.96415 -- received objective: 9.93599 [00020] -- best objective: 9.96415 -- received objective: 9.35591 [00021] -- best objective: 9.96415 -- received objective: 9.90210 [00022] -- best objective: 9.97627 -- received objective: 9.97627 Objective has improved from 9.96415 -> 9.97627 [00023] -- best objective: 9.98883 -- received objective: 9.98883 Objective has improved from 9.97627 -> 9.98883 [00024] -- best objective: 9.98883 -- received objective: 9.97969 [00025] -- best objective: 9.98883 -- received objective: 9.96051 [00026] -- best objective: 9.98883 -- received objective: 9.86835 [00027] -- best objective: 9.98883 -- received objective: 9.80940 [00028] -- best objective: 9.98883 -- received objective: 9.84498 [00029] -- best objective: 9.98883 -- received objective: 9.86562 [00030] -- best objective: 9.99664 -- received objective: 9.99664 Objective has improved from 9.98883 -> 9.99664 [00031] -- best objective: 9.99664 -- received objective: 9.99541 [00032] -- best objective: 9.99790 -- received objective: 9.99790 Objective has improved from 9.99664 -> 9.99790 [00033] -- best objective: 9.99790 -- received objective: 9.99640 [00034] -- best objective: 9.99790 -- received objective: 9.98190 [00035] -- best objective: 9.99790 -- received objective: 9.98854 [00036] -- best objective: 9.99790 -- received objective: 9.98335 [00037] -- best objective: 9.99790 -- received objective: 9.99303 [00038] -- best objective: 9.99790 -- received objective: 9.99271 [00039] -- best objective: 9.99790 -- received objective: 9.99164 [00040] -- best objective: 9.99790 -- received objective: 9.99313 [00041] -- best objective: 9.99790 -- received objective: 9.99236 [00042] -- best objective: 9.99875 -- received objective: 9.99875 Objective has improved from 9.99790 -> 9.99875 [00043] -- best objective: 9.99875 -- received objective: 9.99735 [00044] -- best objective: 9.99969 -- received objective: 9.99969 Objective has improved from 9.99875 -> 9.99969 [00045] -- best objective: 9.99969 -- received objective: 9.99755 [00046] -- best objective: 9.99969 -- received objective: 9.99742 [00047] -- best objective: 9.99995 -- received objective: 9.99995 Objective has improved from 9.99969 -> 9.99995 [00048] -- best objective: 9.99995 -- received objective: 9.99725 [00049] -- best objective: 9.99995 -- received objective: 9.99746 [00050] -- best objective: 9.99995 -- received objective: 9.99990 [00051] -- best objective: 9.99995 -- received objective: 9.99915 [00052] -- best objective: 9.99995 -- received objective: 9.99962 [00053] -- best objective: 9.99995 -- received objective: 9.99930 [00054] -- best objective: 9.99995 -- received objective: 9.99982 [00055] -- best objective: 9.99995 -- received objective: 9.99985 [00056] -- best objective: 9.99995 -- received objective: 9.99851 [00057] -- best objective: 9.99995 -- received objective: 9.99794 Stopping the search because it did not improve for the last 10 evaluations!

Tutorials

  • [NEW] Hyperparameter search for text classification (Pytorch)
  • [NEW] Neural Architecture Search with Multiple Input Tensors
  • [NEW] From Neural Architecture Search to Automated Deep Ensemble with Uncertainty Quantification
  • [UPDATED] Execution on the Theta supercomputer/N-evaluation per 1-node

Hyperparameter search

  • [NEW] Filtering duplicated samples: New parameters filter_duplicated and n_points appeared for deephyper.search.hps.AMBS. By default filter_duplicated = True implies that the search space filters duplicated values until it cannot sample new unique values (and therefore will re-sample existing configurations of hyperparameters). This filtering behaviour and sampling speed are sensitive to the n_points parameter which corresponds to the number of samples drawn from the search space before being filtered by the surrogate model. By default n_points = 10000. If filter_duplicated = False then the filtering of duplicated points will be skipped but n_points will still impact sampling speed.
  • Arguments of AMBS were adapted to match the maximisation setting of DeepHyper: "LCB" -> "UCB", cl_min -> cl_max, "cl_max" -> "cl_min".

Neural architecture search

The package deephyper.nas was restructured. All the neural architecture search space should now be subclasses of deephyper.nas.KSearchSpace:

```python import tensorflow as tf

from deephyper.nas import KSearchSpace from deephyper.nas.node import ConstantNode, VariableNode from deephyper.nas.operation import operation, Identity

Dense = operation(tf.keras.layers.Dense) Dropout = operation(tf.keras.layers.Dropout)

class ExampleSpace(KSearchSpace):

def build(self):

    # input nodes are automatically built based on `input_shape`
    input_node = self.input_nodes[0] 

    # we want 4 layers maximum (Identity corresponds to not adding a layer)
    for i in range(4):
        node = VariableNode()
        self.connect(input_node, node) 

        # we add 3 possible operations for each node
        node.add_op(Identity())
        node.add_op(Dense(100, "relu"))
        node.add_op(Dropout(0.2))

        input_node = node

    output = ConstantNode(op=Dense(self.output_shape[0]))
    self.connect(input_node, output)

    return self

space = ExampleSpace(inputshape=(1,), outputshape=(1,)).build() space.sample().summary() ```

will output:

```console Model: "model_1"

Layer (type) Output Shape Param #

input_0 (InputLayer) [(None, 1)] 0

dense_3 (Dense) (None, 100) 200

dense_4 (Dense) (None, 100) 10100

dropout_2 (Dropout) (None, 100) 0

dense_5 (Dense) (None, 1) 101

Total params: 10,401 Trainable params: 10,401 Non-trainable params: 0

```

To have a complete example follow the Neural Architecture Search (Basic) tutorial.

The main changes were the following: * AutoKSearchSpace, SpaceFactory, Dense, Dropout and others were removed. Operations like Dense can now be created directly using the operation(tf.keras.layers.Dense) to allow for lazy tensor allocation. * The search space class should now be passed directly to the NaProblem.search_space(KSearchSpaceSubClass). * deephyper.nas.space is now deephyper.nas * All operations are now under deephyper.nas.operation * Nodes are now under deephyper.nas.node *

Documentation

  • API Reference: A new section on the documentation website to give details about all usable functions/classes of DeepHyper.

Suppressed

  • Notebooks generated with deephyper-analytics were removed.
  • deephyper ray-submit
  • deephyper ray-config
  • Some unused dependencies were removed: balsam-flow, deap.

Scientific Software - Peer-reviewed - Python
Published by Deathn0t about 4 years ago

DeepHyper - Changelog - DeepHyper 0.3.0

This new release help us move toward a more stable version of DeepHyper.

  • Refactored the DeepHyper Documentation
  • Developed notebook tutorials
  • Decoupled the command line and Python interfaces
  • Refactored the Evaluator interface with evaluator.submit/gather
  • Added deephyper.ensemble for ensembles with uncertainty quantification
  • Removed deephyper.post

Scientific Software - Peer-reviewed - Python
Published by Deathn0t about 4 years ago

DeepHyper - Changelog - DeepHyper 0.2.5

General

Full API documentation

The DeepHyper API is now fully documented at DeepHyper API

Tensorflow-Probability as a new dependency

TensorFlow Probability is now part of DeepHyper default set of dependencies

Automated submission with Ray at ALCF

It is now possible to directly submit with deephyper ray-submit ... for DeepHyper at the ALCF. This feature is only available on ThetaGPU for now but can be extended to other systems by following this script.

ThetaGPU at ALCF

New documentation for auto-sklearn search with DeepHyper

The access to auto-sklearn features was changed to deephyper.sklearn and a new documentation is available for this feature at User guide: AutoSklearn

New command lines for DeepHyper Analytics

The deephyper-analytics command was modified and enhanced with new features. The see the full updated documentation follow DeepHyper Analytics Tools.

The topk command is now available to have quick feedback from the results of an experiment: $ deephyper-analytics topk combo_8gpu_8_agebo/infos/results.csv -k 2 '0': arch_seq: '[229, 0, 22, 1, 1, 53, 29, 1, 119, 1, 0, 116, 123, 1, 273, 0, 1, 388]' batch_size: 59 elapsed_sec: 10259.2741303444 learning_rate: 0.0001614947 loss: log_cosh objective: 0.9236862659 optimizer: adam patience_EarlyStopping: 22 patience_ReduceLROnPlateau: 10 '1': arch_seq: '[229, 0, 22, 0, 1, 235, 29, 1, 313, 1, 0, 116, 123, 1, 37, 0, 1, 388]' batch_size: 51 elapsed_sec: 8818.2674164772 learning_rate: 0.0001265946 loss: mae objective: 0.9231553674 optimizer: nadam patience_EarlyStopping: 23 patience_ReduceLROnPlateau: 14

Neural architecture search

New documentation for the problem definition

A new documentation for the neural architecture search problem setup can be found here.

It is now possible to defined auto-tuned hyperparameters in addition of the architecture in a NAS Problem.

New Algorithms for Joint Hyperparameter and Neural Architecture Search

Three new algorithms are available to run a joint Hyperparameter and neural architecture search. The Hyperparameter optimisation is defined as HPO and neural architecture search as NAS.

  • agebo (Aging Evolution for NAS with Bayesian Optimisation for HPO)
  • ambsmixed (an extension of Asynchronous Model-Based Search for HPO + NAS)
  • regevomixed (an extension of regularised evolution for HPO + NAS)

A run function to use data-parallelism with Tensorflow

A new run function to use data-parallelism during neural architecture search is available (link to code)

To use this function pass it to the run argument of the command line such as:

console deephyper nas agebo ... --run deephyper.nas.run.tf_distributed.run ... --num-cpus-per-task 2 --num-gpus-per-task 2 --evaluator ray --address auto ...

This function allows for new hyperparameters in the Problem.hyperparameters(...):

python ... Problem.hyperparameters( ... lsr_batch_size=True, lsr_learning_rate=True, warmup_lr=True, warmup_epochs=5, ... ) ...

Optimization of the input pipeline for the training

The data-ingestion pipeline was better optimised to reduce the overheads on GPU instances:

python self.dataset_train = ( self.dataset_train.cache() .shuffle(self.train_size, reshuffle_each_iteration=True) .batch(self.batch_size) .prefetch(tf.data.AUTOTUNE) .repeat(self.num_epochs) )

Easier model generation from Neural Architecture Search results

A new method is now available from the Problem object Problem.get_keras_model(arch_seq) to easily build a Keras model instance from an arch_seq (list encoding a neural network).

Scientific Software - Peer-reviewed - Python
Published by Deathn0t over 4 years ago

DeepHyper - Changelog - DeepHyper 0.2.1

Minor bug corrections

Scientific Software - Peer-reviewed - Python
Published by Deathn0t about 5 years ago

DeepHyper - Changelog - DeepHyper 0.2.0

  • Compatible with Tensorflow 2.
  • Horovod compatibility with Balsam evaluator for Theta.
  • Horovod and Balsam are now optional installations.
  • Update of the AMBS algorithm for Hyperparameter search for better scalability.
  • Removing the PPO search for Neural Architecture Search.
  • Creating the SpaceFactory interface for the deepspace package which provides ready to go neural architecture search spaces.
  • Local distribution of jobs with Ray and multiprocessors CPUs.

Scientific Software - Peer-reviewed - Python
Published by Deathn0t about 5 years ago

DeepHyper - DeepHyper 0.1.13

DeepHyper 0.1.13

New NAS Algorithm

  • Aging Evolution with Bayesian Optimization (AgEBO)

New AMBS implementation

  • Previous AMBS renamed to ambsv1
  • New implementation of AMBS for better scaling capabilities

Data-Parallelism settings for Balsam and Horovod

Graph convolution layers with message passin

Scientific Software - Peer-reviewed - Python
Published by Deathn0t about 5 years ago

DeepHyper - 0.1.12

A release for the creation of a DOI on Zeno.

Scientific Software - Peer-reviewed - Python
Published by Deathn0t about 5 years ago

DeepHyper - 0.1.2

Changelog - DeepHyper 0.1.2

DeepHyper 0.1.2 is now forward-compatible with Python 3.7+ and Balsam 0.3.8+ after removing the async reserved keyword.

Scientific Software - Peer-reviewed - Python
Published by Deathn0t about 6 years ago

DeepHyper - 0.1.1

Changelog - DeepHyper 0.1.1

This release is mostly introducing features for Neural Architecture Search with DeepHyper.

DeepHyper command-line interface

For hyperparameter search use deephyper hps ... here is an example for the hyperparameter polynome2 benchmark:

bash deephyper hps ambs --problem deephyper.benchmark.hps.polynome2.Problem --run deephyper.benchmark.hps.polynome2.run

For neural architecture search use deephyper nas ... here is an example for the neural architecture search linearReg benchmark:

bash deephyper nas regevo --problem deephyper.benchmark.nas.linearReg.Problem

Use commands such as deephyper --help, deephyper nas --help or deephyper nas regevo --help to find out more about the command-line interface.

Create an Operation from a Keras Layer

  • Create a new Operation directly from tensorflow.keras.layers:

```python

import tensorflow as tf from deephyper.search.nas.model.space.node import VariableNode from deephyper.search.nas.model.space.op import Operation vnode = VariableNode() vnode.add_op(Operation(layer=tf.keras.layers.Dense(10))) ```

Trainer default CSVLogger callback

  • TrainerTrainValid now has a default callback: tf.keras.callbacks.CSVLogger(...)

Ray evaluator

The ray evaluator is now available through ... --evaluator ray... for both hyperparameter and neural architecture search.

Seeds for reproducibility

To use a seed for any run do Problem(seed=seed) while creating your problem object.

AMBS learner distributed

Use the .. --n-jobs ... to define how to distribute the learner computation in AMBS.

MimeNode to replicate actions

The goal of MimeNode is to replicate the action applied to the targeted variable node.

```python import tensorflow as tf

from deephyper.search.nas.model.space.node import VariableNode, MimeNode from deephyper.search.nas.model.space.op.op1d import Dense

vnode = VariableNode() dense10op = Dense(10) vnode.addop(dense10op) vnode.addop(Dense(20))

mnode = MimeNode(vnode) dense30op = Dense(30) mnode.addop(dense30op) mnode.addop(Dense(40))

The first operation "Dense(10)" has been choosen

for the mimed node: vnode

vnode.set_op(0)

assert vnode.op == dense10op

mnode is miming the choice made for vnode as you can see

the first operation was choosen as well

assert mnode.op == dense30op ```

MirrorNode to reuse the same operation

The goal of MirroNode is to replicate the action applied to the targeted VariableNode, ConstantNode or MimeNode.

```python import tensorflow as tf

from deephyper.search.nas.model.space.node import VariableNode, MirrorNode from deephyper.search.nas.model.space.op.op1d import Dense

vnode = VariableNode() dense10op = Dense(10) vnode.addop(dense10op) vnode.addop(Dense(20))

mnode = MirrorNode(vnode)

The operation "Dense(10)" is being set for vnode.

vnode.set_op(0)

The same operation (i.e. same instance) is now returned by both vnode and mnode.

assert vnode.op == dense10op assert mnode.op == dense10op ```

Tensorboard and Beholder callbacks available for post-training

Tensorboard and Beholder callbacks can now be used during the post-training. Beholder is a Tensorboard which enable you to visualize the evolution of the trainable parameters of a model during the training.

python Problem.post_training( ... callbacks=dict( TensorBoard={ 'log_dir':'tb_logs', 'histogram_freq':1, 'batch_size':64, 'write_graph':True, 'write_grads':True, 'write_images':True, 'update_freq':'epoch', 'beholder': True }) )

Scientific Software - Peer-reviewed - Python
Published by Deathn0t over 6 years ago