Difference: SMACTaskList (100 vs. 101)

Project Goals

HAL"> SMAC in HAL

HAL through Java/MA"> Port SMAC to HAL through Java/MATLAB interface (January)

• Implement: ROAR as instantiation of SMBO framework (FH: 8h)
• Implement: Get Matlab interface going (FH: 4h + 3h for Mav. to read up)
• Implement: EIC optimization (using models to select parameter configurations)
  • Simple random sampling: (FH: 4h)
  • Local search: (FH: 8h) (b/c interface with HAL)
• Reproduce all experiments from SMAC paper within HAL through API
• Reproduce some experiments from SMAC paper through web GUI, to make sure it works

Pure Java implementation of SMAC (January -> April)

• Implement Random Forests in Java
• Implement PCA in Java

HAL"> Gathering runtime data from HAL (January -> February)

• e.g. CPLEX, Spear, SAT data sets

HAL (February - Marc"> Analysis in HAL (February -> March)

• Bootstrap & permutation tests in R and Java (1d/wk)

Prioritized Task List

• HAL UI page for SMAC
• Code clean-up / code review for SMAC --- 3 hours Thursday March 10
• Make inputs to ExperimentRunner actual inputs to the jar --- 1 day
• Enable users to just run SMAC (similar to ParamILS), with SMAC still using HAL under the hood

• Parameterize SMAC (--- 3 hrs)
• Random Forests in Java
• PCA in Java

• Edit wrapper scripts (Which ones?)

• Feature: start SMAC from previous run-logs

• Implement capping in ROAR (need CN and FH)
• Fake configuration scenarios (need CN advice and FH)

Weekly Goals/Milestones

May 30 - Jun 3

• GPGO
  • Partial results -> do postprocessing

• • Run gpgo
• ParamILS comparison to smac.

May 23 - May 27

• Start experiments with Surrogate Algorithms

May 2 - May 6

• Continued from last week.
• BBOB http://coco.gforge.inria.fr/doku.php?id=bbob-2010-downloads
• Call HAL from matlab in order to do target runs. (Look at paramils code)

Apr 26 - Apr 29

• Continued from last week.

1. SMAC UI
2. Implement Surrogate Algorithm
3. Gathering data:
   1. Collect existing runtime matrices ii. Generate new runtime matrices from Westgrid

Apr 18 - Apr 22

1. Surrogate Configuration Scenarios
   1. Algorithm runs on Westgrid Random configurations + "good configurations" (See Frank's thesis: 10.2.1)
   2. Implement surrogate algorithm
2. SMAC UI with CF
   1. Use FeatureInstanceMetricMetaProblemInsatnce in SMAC code.

Apr 11 - Apr 15

• Surrogate Configuration Scenarios:
  • Read up on SATLIB, TSPLIB, MIPLIB, ORLIB
  • BBOB
    • interface with test problems?

• Debugging
  • Being able to start from a previous state for SMAC

Apr 4 - Apr 8

• Look into the #CplexBug
• Debugging SMAC
  • make RunHistory serializable. (What else needs to be serialized?)
• "Fake Configuration Scenarios" - Surrogate Algorithms
• Results for analysis plugin (in progress...):
  1. Look into BCI bug in RStatistics
  2. Compare search steps instead of runtime
  3. Sanity check with different wp/noise parameter.

Mar 28 - Apr 1

• Goal: Results for analysis by Friday, April 8th
  • Implemented additional outputs (from SCD) and confidence intervals for each output
  • Fixed bug with termination
  • Comparison for Walksat
    • Wrappers for ubcsat and original (Technically, I don't need ParamILS wrappers because I'll be using HAL...)
    • Bug: wrappers aren't terminated.
    • Bug: null pointer exception in RStatistics.
    • Experimental runs
• Compile jar for (old) ActiveConfigurator
• (Old) Paper results:
  • /ubc/cs/research/arrow-raid1/hutter/LION5/k_pils-mult-cat/
  • .../k_pils-single-cat
  • .../N_SMBO_mult for ROAR and SMAC
  • -c in the latter ones means continuous
  • .../N_SMBO_single for the single instance scenarios
• Debugging Bootstrap CI plugin
• Debugging SMAC
  • Log whether a selection was random, ei.random, ei.local search.
  • Keep statistics on above.
  • Output callstring to smac.jar into the runlog file.
  • Be able to start smac from a runlog
• "Cplex bug"
  • Test performance for configurations found by ParamILS (when run through HAL) doesn't match results from paper.
  • I need CPLEX 10.1
• Features for runner:
  • Start from list of instances instead of a directory.
    • (First implementation done, but how to handle relative paths, etc?)
• Implement UI page for SMAC (with features).

Mar 21 - Mar 25

• Implement features for Dave:
  • loading from ParamILS files
  • trajectory file
  • instance lists
  • bugfixing

Mar 14 - Mar 18

• Commandline runner that works with Frank's scripts.

Mar 7 - Mar 11

• SMAC logging - log into the same directory as the results. (Estimate: 1 hr)
• Fix bug in new SMAC version--?

Feb 28 - Mar 4

• Random Forests (Estimate: 1 week?)
• Feature loading in HAL (Estimate: 2 days)
  • testing with fake feature extractors on dummy instances
  • More extensive testing with "real" feature extractors on real instances (Est: 1 day)
  • Use "K-way comparison" to run all feature extractors (not necessarily faster?)
• Matlab output/graphs (Estimate: 3 days or when runs finish without exceptions)
• (Tues) EARG presentation
• (Thurs) CO-OP Site Visit

Feb 21 - Feb 25

• Preparation for Mar 1 EARG
• Random Forests (Estimate: 2 weeks)

Feb 14 - Feb 18

• SMAC Logging

Feb 7 - Feb 11

• Investigate MCR Licenses
• Rewrite "output" code for runner
• BUGFIX: Matlab model fails with mixed Integer/Double parameter values (CN)
• Re-write Experiment Runner to poll DB instead of storing runs.
• Delay: experimentrunner exceptions when running on cluster
• Ask DB for instance distributions

Jan 31 - Feb 4

• Experiment Script:
  • Incorporate HAL changes (immutable meta-problem instances)
• Unit Tests:
  • Write Unit test that fails for NeighbourhoodBuilder (handling Numerical Domain sampling)
  • Unit tests for Selector, Model, Neighbourhood Builder

Jan 24 - Jan 28

• Start BootstrapCI/PermutationTest plugin for HAL
• Westgrid
  • Westgrid JavaBuilder Hello World (lattice, orcinus)
• Matlab Model (dependence on FH)
  • Move code for reading instance features / building instance list from MatlabRfModel into AbstractModel
  • Drop instances for which features can't be read.
• Ask DB for instance distributions
• Build smac.jar plugin for HAL (Goal: Jan 26 / dependence on FH)
• Wiki
  • HAL WebUI Guide
  • ExperimentRunner Guide
• Unit test for model building.

Jan 17 - Jan 21

• Note: when blocking, send email to Frank and move onto next item. Not expected to finish all tasks this week.
• Priority 1: Implement: model and selector for SMAC via interface with Matlab code cont. from last week
  • Generate smac.jar plugin for website (have James/ChrisF test), dependence on FH
• Priority 2: Experiments and experiment setup (from last week)
  • Write script to run experiments from smac paper with FH
    • Wiki for steps to run experiments (new item)
    • Run CPLEX, SPEAR, SAT instances with GGA, ParamILS, ROAR
    • output results (configuration, performance) to filesystem
    • Also, format for matlab plotting
• Priority 3: HAL
  • WebUI Guide and reorganize tutorials (this week)
  • Enable better bugfixing by having HAL output the command used to run wrapper (feature is in newest HAL)
  • For convenience of users: output line for calling wrapper with final incumbent (with James)
• Priority 4: Java/Matlab "Hello World" on Westgrid
• Priority 5: Be able to get runtime data from HAL in the same format as webpage below (1000 random configurations with #instances runs each) refer to Hutter 2009 p.61, 225
  • Read Ch.4 in FH thesis and http://www.cs.ubc.ca/labs/beta/Projects/AAC/empirical_analysis/index.html
• Priority 5: Be able to write/read parameter configurations, runtimes per instance indexed by configuration from Java using HAL parameter settings (see cfg file) (this week or next week)
• Priority 5: Edit (.rb) wrapper scripts to allow HAL to terminate it early and delete temporary files; and return exit code for failure (Ask CF)
• (Jan 17) Working on matlab model code in Java.
• (Jan 18) Debugging Java code (model, selector) -- now runs locally on single instance.
• (Jan 18) model: Started dealing with instance features. Started .csv reader.
• (Jan 19) Reviewed and sent pdfscan. Finished csv reader. Started reworking SMBORunner to take CLI args.
• (Jan 20) Reading instance features implemented.
• (Jan 21) Debugging model, selector, runner. Need to change paths in "wrapper" scripts.

Jan 10 - Jan 14

• ROAR Experiments and experiment setup (Goal: Jan 12)
  • Set up
  • Note: Unpredicted overhead for debugging : 1 day
  • Write script to run experiments from HAL paper with FH- Production experiments (moved to next week)
    • Small test run with single instance, uncollected results (locally Jan 11, beta Jan 12, westgrid Jan 13)
    • Run CPLEX, SPEAR, SAT instances with GGA, ParamILS, ROAR and
    • output results (configuration, performance) to filesystem (in progress)
• Register for Westgrid access (done Jan 13)
  • Lattice access?
• Implement: model and selector for SMAC via interface with Matlab code (Goal: next week / in progress)
  • Done: implemented EI-Optimizing selector without local search (Jan. 14)

Jan 4 - Jan 7

• Getting started.
  • Set up Eclipse and git
  • password-less ssh
  • Reading the HAL paper: done first pass
  • Reading the SMAC paper: done
  • Reading the Algorithm Configuration chapter outstanding: pdfscan typos (Jan 19)
• Refactor ROAR to SMBO framework with FH
  • Interfaces for Selector, Model
• Test runs with dummy algorithm on local machine, arrow, and beta

BBOB

• BBOB Code: http://coco.gforge.inria.fr/doku.php?id=bbob-2010-downloads
• /ubc/cs/project/arrow/hutter/bbob/bbob.v10.2/matlab

CPLEX bug infodump

• Use the correct CPLEX version and wrapper: CPLEX 10.1 CPLEX 10 HAL Wrapper (altuning/algorithms/cplex/hal_cplex10_wrapper)
• SMAC paper results

• Step 1: validation runs

/ubc/cs/project/arrow/hutter/ParamILS/cplex/CATS100-first1000-strat-det_algo/paramils-out

You only need to be concerned about these files: focused-dec29-runs2000-runobjruntime-overallobjmean10-bm2-time5.0-tunerTime18000.0-algocplex-*

For example, test performance for the first run is in focused-dec29-runs2000-runobjruntime-overallobjmean10-bm2-time5.0-tunerTime18000.0-algocplex-test_0.txt

The corresponding configuration is at the end of focused-dec29-runs2000-runobjruntime-overallobjmean10-bm2-time5.0-tunerTime18000.0-algocplex-log_0.txt

Paper results for ParamILS

Update

• It seems like the issue was with using CPLEX12 instead of CPLEX10, causing the validated performances to worsen.

Permutation test notes

• GENERAL PROCEDURE FOR PERMUTATION TESTS To carry out a permutation test based on a statistic that measures the size of an effect of interest:

1. Compute the statistic for the original data.
2. Choose permutation resamples from the data without replacement in a way that is consistent with the null hypothesis of the test and with the study design. Construct the permutation distribution of the statistic from its values in a large number of resamples.
3. Find the P-value by locating the original statistic on the permutation distribution.

• Permutation test for matched pairs. The key step in the general procedure for permutation tests is to form permutation resamples in a way that is consistent with the study design and with the null hypothesis. Our examples to this point have concerned two-sample settings. How must we modify our procedure for a matched pairs design?

Ongoing Tasks

• split: HowToGuides HALWebTutorial
• Wiki

-- Main.mavc - 05 Jan 2011