SMAC: Sequential Model-based Algorithm Configuration

News

Abstract

SMAC (sequential model-based algorithm configuration) is a versatile tool for optimizing algorithm parameters (or the parameters of some other process we can run automatically, or a function we can evaluate, such as a simulation).
SMAC has helped us speed up both local search and tree search algorithms by orders of magnitude on certain instance distributions. Recently, we have also found it to be very effective for the hyperparameter optimization of machine learning algorithms, scaling better to high dimensions and discrete input dimensions than other algorithms. Finally, the predictive models SMAC is based on can also capture and exploit important information about the model domain, such as which input variables are most important.
We hope you find SMAC similarly useful. Ultimately, we hope that it  helps algorithm designers focus on tasks that are more scientifically valuable than parameter tuning.

Software

• SMAC version 2.10.03 (July 2015) [source, executables, documentation, examples] (tar.gz, 13,799KB).
  New format for specifying parameter spaces, which has better support for conditionals and forbidden parameters. Improved memory usage, and other bug fixes. License change to AGPLv3.
  The zip file includes a rewritten Quickstart guide, an extensive Manual, and a FAQ.
  

Previous Versions

• SMAC version 2.10.02 (July 2015) [source, executables, documentation, examples] (tar.gz, 13,784KB).
  New format for specifying parameter spaces, which has better support for conditionals and forbidden parameters. Improved memory usage, and other bug fixes. License change to AGPLv3.
  The zip file includes a rewritten Quickstart guide, an extensive Manual, and a FAQ.
  
• SMAC version 2.08.00 (August 2014) [source, executables, documentation, examples] (tar.gz, 12,887KB).
  Major usability improvements see change log.
  The zip file includes a rewritten Quickstart guide, an extensive Manual, and a FAQ.
  
• SMAC version 2.06.01 (October 2013) [source, executables, documentation, examples] (tar.gz, 10,407KB).
  Minor bug fixes:
  1. Fixed bug introduced in v2.06.00 that caused the training set to be validated instead of the test set
  2. SMAC will now try and save a state file in case it is being shutdown by the user or an Error has occurred
  
  The zip file includes a Quickstart guide, an extensive Manual, and a FAQ.
  
• SMAC version 2.06.00 (August 2013) [source, executables, documentation, examples] (tar.gz, 10,366KB).
  Significant Usability Improvements
  Previously a BETA release was made and that is still available here [source, executables, documentation, examples] (tar.gz, 10,228KB).
  
• SMAC version 2.04.02 (August 2013) [source, executables, documentation, examples] (tar.gz, 10,798KB).
  Third bug fix release see manual for change log.
  
• SMAC version 2.04.01 (February 2013) [source, executables, documentation, examples] (tar.gz, 10,818KB).
  Second bug fix release with minor fixes see manual for change log.
• SMAC version 2.02 (October 2012) [source,executables,documentation, examples] (tar.gz, 9,715KB).
  A complete rewrite of SMAC in Java. This features numerous new features, is well documented, and is portable & easy to use, only relying on Java.
  The zip file includes a Quickstart guide, an extensive manual, and a FAQ.
• SMAC-BBOB [source, executables and examples] (tar.gz, 10.1MB). A SMAC version for expensive continuous blackbox optimization; it uses GPs with a Matern kernel, no initial design, and optimizes EI by a combination of local search, DIRECT, and CMA-ES (see BBOB-2013 workshop paper above). Here is the data underlying that paper: [data] (tar.gz, 136MB);
  [data only for SMAC-BBOB and CMA-ES, 100D] (tar.gz, 11MB)
• d-SMAC version 0.8 [source, executables and examples] (tar.gz, 3.4MB). A distributed version of SMAC (see LION 6 paper above).
• SMAC version 1.0.1 [source, executables and examples] (tar.gz, 5.5MB). Minor bugfix release. 
  Modified version that does not rely on MEX files: version 1.0.1b [source, executables and examples]
  You can run a simple example in Matlab by using the following commands: cd source; addpath(genpath(pwd)); sample_call_from_matlab
  You can then adapt sample_call_from_matlab.m to optimize your own function ...
• SMAC version 1.0 [source, executables and examples] (tar.gz, 4.8MB).
  This is the publically released version of SMAC. SMAC will be free to use for non-commercial purposes.
  There is not yet a quick start guide, but the input format is very similar to that of ParamILS (in fact, SMAC should be able to read ParamILS scenarios files directly, although performance will improve if you also specify features for your problem instances). You can call smbo_runner without any arguments to see some usage examples (the same as we had for ParamILS).
  To reproduce the SMAC version from the LION-5 paper, use the following callstring: smbo_runner <numRun> model rf scenario_file <scenario_file>
  To reproduce the ROAR version from the LION-5 paper, use the following callstring: smbo_runner <numRun> method pure-random scenario_file <scenario_file>

License

Forum

For any comments, questions or concerns please check out the SMAC forum available here

People

• Frank Hutter, Research Group Lead (eq. Assistant Professor), Freiburg University
• Holger Hoos, Professor (UBC)
• Kevin Leyton-Brown, Associate Professor (UBC)
• Kevin Murphy, Google Inc. and Adjunct Professor (UBC)
• Steve Ramage, M.Sc. Student (UBC)

Papers

• Frank Hutter, Holger Hoos, and Kevin Leyton-Brown.
  An evaluation of sequential model-based optimization for expensive blackbox functions
  In GECCO 2013 Blackbox Optimization Benchmarking workshop (BBOB'13). To appear. [pdf][bib]

• Frank Hutter, Holger Hoos, and Kevin Leyton-Brown.
  Parallel Algorithm Configuration
  In: Learning and Intelligent Optimization (LION 6). To appear [pdf][pptx slides][pdf slides][bib]

• Frank Hutter, Holger Hoos, and Kevin Leyton-Brown.
  Bayesian Optimization With Censored Response Data
  2011 NIPS workshop on Bayesian Optimization, Experimental Design, and Bandits. [pdf] [poster] [bib]

• Frank Hutter, Holger Hoos, and Kevin Leyton-Brown.
  Sequential Model-Based Optimization for General Algorithm Configuration
  In LION-5, 2011. [pdf] [ppt slides] [pdf slides] [bib]
  Second best paper prize
  An extended version with additional details is available as UBC tech report TR-2010-10. [pdf] [bib]
  
  
• Frank Hutter, Holger Hoos, Kevin Leyton-Brown, and Kevin Murphy.
  Time-Bounded Sequential Parameter Optimization
  In LION4, 2010 [pdf ] [bib] [slides] 
  Runner-up for the best paper award
  The original publication is available at www.springerlink.com.
  

Data