Workshop Description

While experimental approaches are well established and have reached a high level of sophistication in other sciences, like physics and biology, the types of empirical analysis done in AI often are often rather rudimentary. This workshop aims to improve this situation.

The workshop will bring together researchers from different areas of AI and Operations Research, such as constraint satisfaction and satisfiability, knowledge acquisition, machine learning and neural networks, theorem proving, planning and scheduling, learning, robotics, natural language processing and speech recognition, vision, and many others where empirical methods are used to analyse and evaluate algorithms and systems.

Workshop topics will include:

• the design of computational experiments
• performance criteria for algorithms or systems
• methods for analysing, characterising, and comparing algorithmic performance
• the choice of problem sets - benchmark problems, artificially generated problems, etc.
• the role of theoretical models in experiments
• studies from various fields of AI which exemplify good empirical methodology and its value for obtaining interesting results.
• the ethics of experimentation; e.g., should reports provide enough information for experimental replication? Are AI experimental standards distinct from those of other disciplines?
• to what extent are empirical methods really necessary in AI (possibly due to the inadequacy or impossibility of theoretical methods)? How is AI different from other areas of Computer Science in this respect?

This half-day workshop is organised by Holger H. Hoos and Thomas Stützle; it is co-ordinated with the ECAI-2000 Tutorial "Stochastic Search Algorithms" (presented by Holger H. Hoos and Thomas Stützle) and the ECAI-2000 Tutorial "Empirical Methods for Computer Science" (presented by Paul Cohen, Ian Gent, and Toby Walsh). The workshop and the two tutorials will complement each other and together will offer a comprehensive coverage of many important issues in the empirical analysis of AI algorithms and systems.

Important Dates

Submission Procedure

We ask authors to submit a position paper either in Postscript or PDF.

The position paper should describe and justify your view on one or more topics relevant to this workshop. The paper should be reasonably concise, i.e. we expect that about 1000 words will suffice, but you may use more if needed. Submissions should be printed on 8.5" x 11" or A4 paper with at least 1 inch margins on all sides. The first page of the position paper should include the title, a brief abstract, and author names, affiliations, postal addresses, electronic mail addresses, and telephone and fax numbers.

To submit a paper, email it (or a URL pointer to it) to hoos@cs.ubc.ca or stuetzle@informatik.tu-darmstadt.de.

The number of participants is limited; participants will be selected based on the submitted position paper. All participants are expected to contribute to the discussions in the workshop.

Note that all workshop participants are expected to register for the main ECAI-2000 conference

Organisation / Contact Information