HAL 1.1+ Projects

Memory Efficiency

• redundant instantiation of immutable or largely immutable objects
• inefficient serialized representations of hierarchical/composite objects (currently, data manager manually removes this redundancy for a limited set of objects)
• needed to scale to very large instance sets, deeply nested meta-algorithm instances, etc.

Profiling/Optimizations

• general performance and developer experience improvements

Detection of and Cleanup After Edge Case or Unexpected Behaviour

• e.g. network outages, crashed/hanged/disappeared runs, etc.

Removal of RPC Reliance/Data Manager-centric Communications

• used to implement some AlgorithmRun methods for remote runs (getCpuTime, etc.)
• fragile, and causes noticeable delays in UI
• probably require support for unstructured broadcast communications via data mgr.

SSH Execution Manager improvements

• many limitations when running targets over SSH (See devel guide)
• possibly fix by having dedicated per-machine run controller instance? Possibly related to EDACC-style execution? Also related to removal of shared FS.

Removal of Shared Filesystem Requirement

• transfer of algs, instances, etc.
• monitoring output for remote SSH and Cluster execution managers

Cluster batching support

• needed for lattice support
• useful in general when distributing many short runs over cluster.

Problem-specific, procedure-agnostic UI

• replace/augment "instances" with space-like introspectable representation
• meta-algorithm instance constraints (single target, etc.) should be explicit instead of via tags
• additional UI elements for specific plugins can be bundled in the plugin

Versioning supporting in serialization/deserialization

• e.g., the refactor of metrics implied a change in the serialization format, which broke old exports. It is not clear from the resultant error that this breakage was due to a version change.

Features needing testing

• Overall unit test coverage
• queueRun support for meta-algs via SubrunRunner.
• Reused runs, duplicate experiments w/different metrics, etc.; esp. edge cases w.r.t. runs that (barely) exceed cutoff time
• ParamILS through HAL vs. standalone

HAL 1.0 Task List

Below last updated Dec/2010

For LION

must-haves

• cloning experiments
• run k-times for configuration
• plot data point labels in plot interface (eventually in the UI as well)
• viewing/cloning algorithms, etc.
• refactoring experiment specification pages
• star-Zilla -- demo with Cplex+gurobi? .. using pre-collected data.
• Torque support given arrow-like architecture assumptions
• rudimentary deletion of runs, etc., and a better plan
• screencast of basic HAL usage

medium term

• result data exporting (general run data)
• Westgrid support

Backend Tasks

4-week plan (CN)

• (~1d) test target run distribution w/SGE Exec Mgr (may be working already..)
• (~1d) backend support for user-space textual object annotations.
• (~1d) Outstanding import/export backend issues (exec. environments, object descriptions, etc.)
• (~3d) job health monitoring (check outstanding jobs e.g. on cluster; detect hangs/problems and alter/cleanup appropriately).
• (~1d) sporadic missing finishtime bug

Unprioritized

• (~3d) pause/resume for meta-algorithms [estimate doesn't include pausing target algorithms mid-run, just halting dispatch of new runs]
• (~1d combined) Plotting:
  • incumbent qual plot - extend to current time
  • different colours/linestyles for the two medians in dual SCD plot
  • show cutoff time in plots (dashed lines)
  • different linestyles / labels?? for vertical lines in overlaid SCD plots
• (~2d) deeply lazy database algorithm run objects (mostly, lazy meta problem instances)
• (~2d) decouple toSpec() and getHash(); serialize everything incl. metadata; fix exporting re: this. much less fragile than current situation of an object's hash code depending directly on its serialized representation.
• (~3d??) Problems as predefined semantics; useful mostly for target problems, e.g. planning
• (~1d) Expose more ParamILS features (BasicILS, SQ tuning, etc etc)

HAL-related tasks not c"> Parallel HAL-related tasks not considered in above lists (see below for some details on some of these)

• (???) documentation
• (ongoing) release mgmt
• (??? w/Lin) portfolio-based selection (SATzilla)
• (??? w/James) Improved cluster support (incl. Torque)
  • (1d) assess existing SGE manager on TORQUE, assuming assumptions like common file system are satisfied
• (~1w w/Frank) "virtual" target algorithms (based on Matlab models)
• (??? w/Frank) SMBO (Active Configurator)
  • (??) MatlabBuilder/J integration for SMBO

Frontend tasks

4-week plan (CF)

• (~1d) Landing page run status divs should maintain individual state, not have pieces disappear on refresh, and should not appear under more than one heading (or disappear entirely) when the run status changes.

• (~2d) Algorithm specification screen changes, as per previous HAL meeting.
  • "add output capture" below info for last string added
  • examples for params (call string), output capture

• (~1d) Rearranging meta-algorithm specification pages, as previously discussed.
  • default name
  • ? Submit -> launch
  • after completing exp page, offer choice to
    • launch
    • save as 'ready for launch'
    • ? save as 'in preparation'
  • target algorithm -> algorithm to be analysed
  • hide meta-alg slc (since there is only one choice right now)
  • target algorithm 1/2 -> algorithm 1/2 to be analysed
  • meta-algorithm selected second-but last (just before exec environment)
    • Can't do this. Each meta-algorithm can require different input fields.
  • keep options collapsed by default wherever possible

• (~1.5d) Viewing/editing/copying of HAL top-level objects

• (~1d) Notes/annotation for 'top-level' objects.
  • Algorithm
  • Instance set
  • Experiment
    • On change from partial run to actual run, need to make sure descriptions get transferred.
  • Execution environment
  • Configuration

• (~1d) Various minor previously-reported monitor page changes.
  • auto-reload of monitor pages, interval configurable in HAL settings - local to client/user
  • report HAL overhead also as percentage of overall running time
  • include percentage complete, predicted completion time for experiment
  • single/pairwise analysis: report # / fraction of time-outs
    • adjustment of param names (r) and value (l); 2 columns? scrollable box?
  • Incumbent Design Information: add time when current incumbent was found
    • name for current incumbent / final result? (how can user specify this?)
  • contrast incumbent param settings with default settings (display latter as well)
  • Single alg analysis: refer to alg as "name + (target algorithm)" in monitor/results page
  • Pairwise analysis: refer to alg 1, alg 2 as "name (algorithm 1/2)" in monitor/results page

• (~0.5d) Sub-run information table in the monitor pages, sortable by column headers, not loaded until requested.

• (~0.5d) Database deletion of HAL top-level objects, from the landing page
  • Experiments
  • Ready to launch runs

• (~0.5d) Database deletion of HAL top-level objects, from a separate screen
  • Algorithm
  • Instance set
  • Experiments
  • Ready to launch runs
  • Execution environment
  • Configuration

• (~0.5d) Graying out and showing alg spec. steps or something so that users can see how much they have left.

Unprioritized

• Define New Inst Distr
  • (~0.25d) hide type (until we have > 1 types)

• Define New Alg:
  • (~0.5d) also allow creation of new configs etc. from Create New Alg?
  • ??? easy way to add param domain decl to default (to be copied from there later)
  • ??? alternative ways of specifying param config spaces: text file/text field + parse button?

• ??? number of indep, par runs + auto select best on training. i.e. a new meta-algorithm.

• Alg config results/status:
  • ??? support exporting from inspection pages (for exp, alg, inst sets)

• Export
  • (~0.1d) any -> all?
  • (~0.1d) some labels need to be more intuitive (Filename for Object Export, ...)
  • (~0.1d) size of package should be displayed
  • (0.5d) export instance sets without taking everything in respective directory (checkable option in instance set spec, default = don't take whole dir)

• Import
  • ??? allow user to select within package
  • Allow the user to specify the location of missing files; for now we just warn that they aren't present and import anyway?

• everwhere
  • (mostly covered in 4wk plan) hide (and make expandable) everything that has decent default values most users may not want to touch
  • (~0.25d) capitalisation: do not capitalise 'short words' (yet, not, ...), certain messages ('Not Yet Finished' -> 'not yet finished'), best solution: put all in properties file, let me (hh) edit them.
  • (~0.1d) time & date displays (everywhere): dd-mm-yyyy or yyyy/mm/dd, 24h-times
  • (~0.1d) instance distribution -> instance set
  • (covered mostly in 4wk plan) selection lists with only one choice: make the one choice the default

• (~0.5d) Modify domain transformation interface to accomodate linear transformations / multiple transformations as soon as the backend infrastructure is present.

*zilla+HAL

HAL prep, mostly CN

• (1d) K-way analysis for data collection (no tests yet)
• (1d) Feature extractor as a wrapped external target algorithm
• (1d) "internal" target algorithms (to implement trained selector with; note not a meta-algorithmic procedure per LION def'n)
• (2d) generic "selector" which, given a trained model, extracts features, performs inference, and selects/runs a solver
  • (update: fundamentals done; needs refinement though)

*zilla, w/LX

• (~2d) High-level control in Java
• (~2d) Model building and inference using Matlab from Java
• (~2d) Integration of pieces into complete SATzilla pkg
• (~2d?) UI elements (w CF?)
• (~2d) testing

Hydra+HAL

• plan once *zilla is further along

ActiveConfigurator+HAL

w/FH, ~1-week tasks:

• (2d) Fake configuration scenarios (i.e. fake target algorithms)
• (1d) Update AC project w.r.t. recent HAL changes (mostly FH)
• SMAC
  • (1d) high-level design
  • (2d) impl. using interface to Matlab

New Feature: subsidiary analyses

• User can specify e.g. new metrics for already-performed analyses
• HAL uses an Environment w/new target XM that ONLY uses DB-sourced target runs to quickly compute these modified expts
• Only one modified expt per target expt at a time
• (HH) Think about how to let the user specify plots/etc. at an even finer level of granularity, in more appropriate ways

Old Tasks

-- ChrisNell - 01 Dec 2010