CPSC 532S: Modern Statistical Learning Theory – 2021W2

Instructor: Danica Sutherland (she/her): dsuth@cs.ubc.ca, ICICS X563.
Lecture info: Mondays/Wednesdays, 13:30 - 15:00, DMP 101 on Zoom until February 7.
Office hours: By appointment, for now. I check Piazza a lot; feel free to either just ask your question directly, or make a private post to schedule a meeting.

Schedule

SSBD below refers to the book of Shalev-Shwartz and Ben-David; MRT to that of Mohri, Rostamizadeh, and Talwakar.

1MonJan 10Intro / overviewSSBD chap. 1-2; MRT chap. 2
MonJan 10Assignment 1 posted (and .tex)
2WedJan 12PACSSBD chap. 2-3; MRT chap. 2
3MonJan 17Probability / uniform convergence / more?SSBD chap. 4; MRT chap. 2
4WedJan 19
ThuJan 20Assignment 1 due, 11:59pm
FriJan 21Drop deadline
5MonJan 24
6WedJan 26
7MonJan 31
8WedFeb 2
MonFeb 7Planned shift to hybrid mode (rather than online-only) 🤞
9MonFeb 7
10WedFeb 9
11MonFeb 14
12WedFeb 16
MonFeb 21Midterm break
WedFeb 23Midterm break
13MonFeb 28
14WedMar 2
15MonMar 7
16WedMar 9
17MonMar 14
18WedMar 16
19MonMar 21
20WedMar 23
21MonMar 28
22WedMar 30
23MonApr 4
24WedApr 6

Logistics

The course will initially meet on Zoom: the meeting link is available on Canvas and Piazza. Starting January 24thFebruary 7th, we will hopefully meet in person in DMP 101. I currently plan to both livestream and record lectures throughout the term, either via Zoom (same link) or Panopto (link will be provided if so). Plans here are subject to change.

Recordings are available from both Canvas and Piazza.

Grading scheme: 70% assignments (including a small project), 30% final.

The lowest assignment grade (not including the project) will be dropped. The exact relative weight of assignments and the project is TBD. Assignments should be done in LaTeX – not handwritten or in a word processor. Hand-in procedure will be announced before the first deadline.

There will be one “big assignment” which serves as a (small) project: something on the scale of doing some experiments to explore a paper, doing a lit review in a particular area, extending / unifying a few papers, etc. A proposal will be due beforehand; details to come.

The final exam may be take-home, synchronous online, or in-person; TBD.

There may also be some paper presentations later in the course, in which case the paper presenters will be able to use that to replace part of an assignment grade. This is dependent on the COVID situation and other factors; TBD.

Overview

The brief idea of the course: when should we expect machine learning algorithms to work? What kinds of assumptions do we need to be able to be able to rigorously prove that they will work?

Definitely covered: PAC learning, VC dimension, Rademacher complexity, concentration inequalities. Probably: PAC-Bayes, analysis of kernel methods, margin bounds, stability. Maybe: limitations of uniform convergence, analyzing deep nets via neural tangent kernels, provable gaps between kernel methods and deep learning, online learning, feasibility of private learning, compression-based bounds.

There will be some overlap with CPSC 531H: Machine Learning Theory (Nick Harvey's course, last taught in 2018), but if you've taken that course, you'll still get something out of this one. We'll cover less on optimization / online learning / bandits than that course did, and try to cover some more recent ideas used in contemporary deep learning theory.

(This course is unrelated to CPSC 532S: Multimodal Learning with Vision, Language, and Sound, from Leon Sigal.)

Prerequisites

There are no formal prerequisites. I will roughly assume:

If you have any specific questions about your background, feel free to ask.

Resources

Learning theory textbooks and surveys:

If you need to refresh your linear algebra or other areas of math:

Resources on learning measure-theoretic probability (not required to know this stuff in detail, but you might find it helpful):

Similar courses: