CS340 (Machine learning) Fall 2006

Lectures MWF 4-5, Dempster 301

Instructor: Kevin Murphy

Newsgroup

NEWS UPDATE

28 Nov: hw9 cancelled
21 Nov: I have added Code.zip which contains all the code in the handouts (and lots more!)
9 Nov: here is a list of interesting classes for Spring 07
- Stat406, algorithms for classification and prediction, Murph
- CS406, Optimization, Friedlander
- CS322, AI, Leyton-Brown
- CS420, advanced algorithms, Evans
3 Nov: Hao's discussion section has been replaced by office hours on Fridays, 2-3, X568.
25 Oct: Midterm grades. These are raw percentages, not final marks.
11 Oct: The final exam date will be Mon Dec 11, 3:30-6 pm Dmp 110.

Discussion sections

Choose one from

T1A, 3:00 - 4:00 Thursdays, DMP101
T1B, 8:30 - 9:30 Tuesdays, DMP201

Office hours

Tuesdays, 3-4pm, CS 187

TAs

Textbook

Required (primary textbook):

Introduction to machine learning, Ethem Alpaydin, MIT Press 2004
(70 copies will arrive in UBC bookstore on September 8th)

Pre-requisites

cpsc 221 (data structures and algorithms)
stat 241 (intro prob/stats) or math/stat 302 (intro prob) [241 is recommended]
math 221 (matrix algebra) or math 223 (linear algebra - honors) [223 is recommended]

Syllabus

We will follow Alpaydin's book (for the most part), with some supplemenary handouts. The class will cover supervised learning (classification and regression) and unsupervised learning (density estimation, i.e., clustering, dimensionality reduction). Applications/examples will be drawn from computer vision, computational biology, speech/language, etc. The class will involve a fair amount of math (linear algebra, calculus and basic probability/ statistics), but also a fair amount of hands-on coding (in Matlab).

Grading

Midterm: 30%
Final: 45%
Assignments: 25%

Timetable

Reading material is abbreviated as follows: A = Alpaydin book, B = Bishop book, H = Hastie book.

Lecture notes include material borrowed from a variety of authors. Please contact me before copying.

L# Date Slides Reading Homework

L1 Wed Sep 6
Intro

Optional: Matlab for psychologists by Antonia Hamilton.
Recommended: Matlab hints and tips by Kevin Murphy.
.

L2 Fri Sep 8
Intro A chap 1 .

L3 Mon Sep 11
Binary classification A 2.1, 2.4, 2.5, 2.7, 2.8 HW1: complete these matlab exercises by Mon Sep 18. The data is here. Clarifications

L4 Wed Sep 13
Learning theory A 2.2, A 2.3 .

L5 Fri Sep 15
Learning theory

Learning theory, part of chapter 18 of Artificial Intelligence: a Modern approach, by Stuart Russell and Peter Norvig
.

L6 Mon Sep 18
Bayesian concept learning

Bayesian concept learning, sections 1--5 (sec 6 is optional),
Optional: Theory-based Bayesian models of inductive learning and reasoning, Tenenbaum, Griffiths, Kemp, TICS 2006
HW2 by Mon Sep 25. The data/code is here. HW2 clarifications.

L7 Wed Sep 20
Bayesian concept learning Refresher on probability. .

L8 Fri Sep 22
Bayesian concept learning . .

L9 Mon Sep 25
Concept learning; clustering . HW3 by Mon Oct 2. The data/code is here. HW3 clarifications

L10 Wed Sep 27
Modeling discrete data with Bernoulli and multinomial distributions

A 4.1-4.4
Parameter estimation for bernoullis and multinomials
.

L11 Fri Sep 29
Modeling discrete data with Bernoulli and multinomial distributions Optional: In praise of Bayes, Economist article, Sept 2000 .

L12 Mon Oct 2
Naive Bayes classifiers Naive Bayes classifiers HW4 by Wed Oct 11. Code/data is here.

L13 Wed Oct 4
Naive Bayes classifiers Optional: Better Bayesian [spam] filtering, Paul Graham, Jan 2003 .

L14 Fri Oct 6
Markov models

Markov models
A 13.2
.

L15 Mon Oct 9
Thanksgiving . .

L16 Wed Oct 11
Markov chains . HW5 due Wed Oct 18. Code/data is here.

L17 Fri Oct 13
Information theory Information theory .

L18 Mon Oct 16
Info theory . .

L19 Wed Oct 18
Info theory . No HW!

L20 Fri Oct 20
Review session . .

L21 Mon Oct 23
Midterm . .

L22 Wed Oct 25
Midterm postmortem Notation used so far in class. .

L23 Fri Oct 27
MCMC

MCMC
Optional:An introduction to MCMC for machine learning, Machine Learning J., 50:5--43, 2003
Optional: Top 10 algorithms of the 20th century
Optional: Java MCMC applets
.

L24 Mon Oct 30
MCMC . hw6.pdf, hw6Code.zip,

L25 Wed Nov 1
MCMC . .

L26 Fri Nov 3
MRFs MRFs .

L27 Mon Nov 6
MRFs . .

L28 Wed Nov 8
Simulated annealing (hw 6) . .

L29 Fri Nov 10
Variable elimination . hw7.pdf, hw7Code.zip,

L30 Mon Nov 13
Remembrance day holiday . .

L31 Wed Nov 15
Bayes nets Inference in graphical models .

L32 Fri Nov 17
Applications of Bayes nets Bayes nets hw8.pdf

L33 Mon Nov 20
Plates and param learning A 3.7, Microsoft's mobile manager .

L34 Wed Nov 22
Causality Simpson's paradox (extracted from the bn.pdf file) .

L35 Fri Nov 24
Gaussians Gaussians, A 5.2-5.4 .

L36 Mon Nov 27
Snow day . .

L37 Wed Nov 29
Gaussian mixtures Mixture models, A ch 7 .

L38 Fri Dec 1
Review session . .

- Mon Dec 11
Final exam DMP 110 3.30-6pm .

L#	Date	Slides	Reading	Homework
L1	Wed Sep 6	Intro	Optional: Matlab for psychologists by Antonia Hamilton. Recommended: Matlab hints and tips by Kevin Murphy.	.
L2	Fri Sep 8	Intro	A chap 1	.
L3	Mon Sep 11	Binary classification	A 2.1, 2.4, 2.5, 2.7, 2.8	HW1: complete these matlab exercises by Mon Sep 18. The data is here. Clarifications
L4	Wed Sep 13	Learning theory	A 2.2, A 2.3	.
L5	Fri Sep 15	Learning theory	Learning theory, part of chapter 18 of Artificial Intelligence: a Modern approach, by Stuart Russell and Peter Norvig	.
L6	Mon Sep 18	Bayesian concept learning	Bayesian concept learning, sections 1--5 (sec 6 is optional), Optional: Theory-based Bayesian models of inductive learning and reasoning, Tenenbaum, Griffiths, Kemp, TICS 2006	HW2 by Mon Sep 25. The data/code is here. HW2 clarifications.
L7	Wed Sep 20	Bayesian concept learning	Refresher on probability.	.
L8	Fri Sep 22	Bayesian concept learning	.	.
L9	Mon Sep 25	Concept learning; clustering	.	HW3 by Mon Oct 2. The data/code is here. HW3 clarifications
L10	Wed Sep 27	Modeling discrete data with Bernoulli and multinomial distributions	A 4.1-4.4 Parameter estimation for bernoullis and multinomials	.
L11	Fri Sep 29	Modeling discrete data with Bernoulli and multinomial distributions	Optional: In praise of Bayes, Economist article, Sept 2000	.
L12	Mon Oct 2	Naive Bayes classifiers	Naive Bayes classifiers	HW4 by Wed Oct 11. Code/data is here.
L13	Wed Oct 4	Naive Bayes classifiers	Optional: Better Bayesian [spam] filtering, Paul Graham, Jan 2003	.
L14	Fri Oct 6	Markov models	Markov models A 13.2	.
L15	Mon Oct 9	Thanksgiving	.	.
L16	Wed Oct 11	Markov chains	.	HW5 due Wed Oct 18. Code/data is here.
L17	Fri Oct 13	Information theory	Information theory	.
L18	Mon Oct 16	Info theory	.	.
L19	Wed Oct 18	Info theory	.	No HW!
L20	Fri Oct 20	Review session	.	.
L21	Mon Oct 23	Midterm	.	.
L22	Wed Oct 25	Midterm postmortem	Notation used so far in class.	.
L23	Fri Oct 27	MCMC	MCMC Optional:An introduction to MCMC for machine learning, Machine Learning J., 50:5--43, 2003 Optional: Top 10 algorithms of the 20th century Optional: Java MCMC applets	.
L24	Mon Oct 30	MCMC	.	hw6.pdf, hw6Code.zip,
L25	Wed Nov 1	MCMC	.	.
L26	Fri Nov 3	MRFs	MRFs	.
L27	Mon Nov 6	MRFs	.	.
L28	Wed Nov 8	Simulated annealing (hw 6)	.	.
L29	Fri Nov 10	Variable elimination	.	hw7.pdf, hw7Code.zip,
L30	Mon Nov 13	Remembrance day holiday	.	.
L31	Wed Nov 15	Bayes nets	Inference in graphical models	.
L32	Fri Nov 17	Applications of Bayes nets	Bayes nets	hw8.pdf
L33	Mon Nov 20	Plates and param learning	A 3.7, Microsoft's mobile manager	.
L34	Wed Nov 22	Causality	Simpson's paradox (extracted from the bn.pdf file)	.
L35	Fri Nov 24	Gaussians	Gaussians, A 5.2-5.4	.
L36	Mon Nov 27	Snow day	.	.
L37	Wed Nov 29	Gaussian mixtures	Mixture models, A ch 7	.
L38	Fri Dec 1	Review session	.	.
-	Mon Dec 11	Final exam DMP 110	3.30-6pm	.