You are encouraged to discuss this assignment and collaborate with your classmates, as long as (a) you list the people with whom you discussed the assignment and (b) you give your own explanation for question 1 and choose a different domain for question 2 to your classmates you collaborate with. Feel free to ask questions on the Vista bulletin board.

Question 1

Consider the 6 state, 4 action domain at:

http://artint.info/demos/rl/tGameRand.html

What is the policy that optimizes the accumulated reward? Explain how you found it. (You should do this just by playing with the applet, and trying to work out which actions give the highest accumulated reward).

Question 2

Choose a particular application domain that can be abstracted as an agent, for example, a particular game, the diagnosis and repair of a particular type of object, a robot in a particular environment, a tutoring system in a particular area, or (preferably) some domain you are interested in and knowledgeable about.

Use full sentences in your answers. Explain your answers well enough so that the reader will be able to understand what you mean. If there are no examples explain why.

1. Describe the domain.

2. For a particular type of agent in the domain:

   1. What are its abilities?

   2. What prior knowledge would an agent have?

   3. What observations would an agent make?

   4. What past experiences could an agent learn from?

3. Give an example of a modular decomposition of an agent in your domain.

4. Give an example of a hierarchical decomposition of an agent in your domain.

5. Give examples of features in your domain.

6. Give examples of types of individuals

7. Give examples of relations that are not properties

8. Give examples of what an agent may be uncertain about.

9. Give examples of stochastic actions.

10. Give examples of deterministic actions.

11. Are there subproblems that are fully observable? Explain.

12. Are there subproblems that are partially observable? Explain.

13. Give examples of achievement goals.

14. Give examples of ordinal preferences.

15. Give examples of cardinal preferences.

16. Give a finite horizon problem for this domain.

17. Give a indefinite or infinite horizon problem.

18. How are multiple agents involved? Do they have competing preferences?

19. Could an agent be perfectly rational? Explain.

Question 3

How long did this assignment take? What did you learn? Was it reasonable?

Mini-project suggestions

Remember that everyone need to give a mini-project for January. This involves giving a 3-minute talk to the class, and posting a summary (including references) to WebCT. This should be something that is interesting and informative. Tell us something that you would be interesting in knowing about. Each assignment will give some suggestions.

Find some real-world application, and explain to the class an agent in that domain: what it observes, what it actually does, what background knowledge it has.

At http://artint.info/demos/rl/sGame.html is a simple game with a simplistic controller. All code is open source. Try to build a better controller, and report back on how your controller works and the average reward received. We will use this example for reinforcement learning.