(r13) HAL < BETA

Tags: view all tags
---+ Pre-CRC/release tasks
---++ UI tasks
---+++ Release-critical 
   * *CF* N-way performance comparison first-cut for Frank.
_functionality promised in paper_
   * *CF* algorithm specification screen: implement
   * *CF* left side of landing page:  task selection/presentation according to pattern concept
   * *CF* experiment specification and monitor screens from a pattern template, and procedure-specific requirements (may require DB changes)
   * *CF* instance specification screen: implement (requires DB change)
   * *CF* Execution environment specification (incl. R, Gnuplot, java locations; requires DB change)
   * *CN* RTDs/per-target-algorithm-run monitoring and navigation

---+++ Important
_works as-is but end-user experience significantly impacted_
   * Data management interface:
      * deleting runs/expts/etc.
      * data export
   * Error logging/handling/browsing
   * Plotting ex-gnuplot
   * Named configurations, ability to specify name for final incumbent a priori and reference it in subsequent comparison experiments.


---++ Database tasks
---+++ Release-critical
_mostly to enable critical UI tasks_
   * *CN* Named instance set table *done*
   * *CN* Named configuration table
   * *CN* Execution environment table
   * *CN* Configuration spaces vs. algorithms; appropriate unique ID hashing

---+++ Important
_mostly to (substantially) improve UI responsiveness_
   * Connection pooling
   * Caching analysis results
   * Database schema -- speed-related redesign
   * Query optimization
   * Selective limitation of run-level logging (dynamic based on runtime?)

---+++ Nice-to-have
_noticeable mostly to developer-users_
   * DataManager API refinement
   * Stale connection issue; incl. robustness to general network issues
   * Read-only DataManager connection for use by individual MA procedures


---++ Code/Robustness/Misc. tasks
---+++ Important
   * user-facing documentation (help)
   * Better logging/error-reporting (to console/within HAL).  eg: log4j
   * Better handling of overhead runtime vs. target algorithm runtime
   * WAY more unittests; also functional/integration tests

---+++ Nice-to-have
   * developer-facing documentation (javadocs)


---+ Medium-term Plans
   * Packaging complete experiments
   * Windows support
   * libraries of:
      * search/optimization procedures
      * machine learning tools
   * multi-algorithm comparisons
   * scaling analyses
   * bootstrapped analyses
   * robustness analyses
   * parameter response analyses
   * SATzilla in HAL
   * ParamILS in HAL
   * Parallel portfolios in HAL
   * ActiveConfigurator in HAL
   * Iterated F-Race in HAL
   * chained-procedure experiments
   * support for optimization/Monte-Carlo experiments
   * support instance generators
   * Git, not CVS
   * Support text-file inputs and outputs for external algorithms
   * Instance features
   * Experiments calling experiments, not just external target algs
   * array jobs in SGE
   * Hashing everything, including instances, instance sets and configurations.

---+ Feature Requests
   * (FH) Support for complete configuration experiment, front to back: run configurator N times on a training set, report the N training and test set performances
   * (FH) Developers of configurators should be able to swap in new versions of a configurator
   * (FH) Configuration scenarios, specifying a complete configuration task including the test set; only missing part being the configurator
   * (FH) Saveable sets of configuration scenarios to perform (use case: I change the configurator and want to evaluate it)
   * (FH) Taking this a step further: support for optimizing a parameterized configurator (configurator is an algorithm, and the above set of experiments is the set of "instances")
   * (FH) Submitting runs from a machine that is itelf a cluster submit host should not need to go through SSH
   * (CF) Distinction made in HAL between parameterised and parameter-less algorithms, including the ability to see the "parent" parameterised algorithm of a given parameter-less algorithm. Other similar queries would be equally useful (all SPEAR configurations, the parameterised algorithm corresponding to a partial instantiation of another parameterised algorithm, etc.) _(CN) need to flesh out how this interacts with named configurations/configuration spaces also under discussion_
   * (CF) Ability to browse algorithms, instances, instance sets, configurations, etc. This includes the ability to see things related to the item being browsed. Performance of different algorithms/configurations on a given instance, performance of algorithms across an instance set, performance of a given configuration.
   * (CF) Form validation.
   * (CF) Wider support for working directory requirements of individual algorithm runs, i.e. Concorde's creation of 20 files with fixed names.
   * (CN) Ability to actively manage database, including properly cascaded deletion of elements.

---+ Known Bugs:
   * (CN) JSC test reliability issue (compared to R)
   * (CN) end-of-experiment hanging bug (GGA, multinode cluster runs)
   * (JS) InnoDB SQL errors
   * (LX) missing current-time point in solution quality trace, so don't see the final "flat line"
   * (CN) accuracy of mid-run overhead accounting for PILS/GGA
   * (CF) Configuration file callstrings with weird spaces, i.e. "... -param '$val$ blah' ..." where '$val blah' needs to be passed to the target as a single argument. _(CN) does this work with double-quotes instead of single-quotes?_
   * (JS) FixedConfigurationExperiment UI is outdated, unusable.
   * (JS) HAL is not usable on WestGrid. We need a TorqueClusterExecutionManager.
   * (JS) Algorithms with a requirement of a new directory for each run.
Topic revision: r13 - 2010-05-10 - ChrisNell