Projects for CPSC 532S: Modern Statistical Learning Theory – 2021W2


Projects should be done in groups of at least 1, no more than 3. The project counts as one homework assignment (but can't be dropped).

Project scope

Your paper should fit roughly into one of these three categories:

Some suggestions

Here are a smattering of interesting paper ideas I happen to already know about. It's worth browsing through recent proceedings for COLT (2021, 2020) or ALT (2021, 2020); lots of good learning theory papers appear in other venues, but most papers in these venues are relevant to this course.

Most of these are from the last few years, but it's fine to do older papers too.

Self-supervised learning: theory of pretext tasks, a follow-up; an attempted explanation of contrastive learning; why more negatives don't always help

New models for generalization:via optimal transport; via conditional mutual information; Towards a Unified Information-Theoretic Framework for Generalization; Distributional Generalization

Ensembles:Assessing generalization via disagreement and "a note on" that paper; "deconstructing distributions"; estimating accuracy from unlabeled data

Meta-learning:Provable Guarantees for Gradient-Based Meta-Learning; A Closer Look at the Training Strategy for Modern Meta-Learning

A few disparate applications of kernel mean embeddings: e.g. Distributionally Robust Optimization and Generalization in Kernel Methods; Towards a Learning Theory of Cause-Effect Inference; Learning Theory for Distribution Regression (kind of heavy)

Learning mixtures: sample complexity of mixtures of Gaussians (UBC paper)

Algorithm configuration: this paper, for example; a perspective paper

Fairness and generalization: Recovering from Biased Data: Can Fairness Constraints Improve Accuracy?

Stability of (S)GD: for SGD, followup 1, followup 2 (only partly about SGD)

Neural net generalization based on topology of learning paths

Information-Theoretic Generalization Bounds for Stochastic Gradient Descent

Lottery ticket hypothesis: the main paper; Stabilizing the Lottery Ticket Hypothesis; Linear Mode Connectivity and the Lottery Ticket Hypothesis; Pruning Neural Networks at Initialization: Why are We Missing the Mark?

Domain generalization:Invariant Risk Minimization, The Risks of Invariant Risk Minimization, Does Invariant Risk Minimization Capture Invariance? (extra points for sucking up if you pick this last one) (not really); Measuring Robustness to Natural Distribution Shifts in Image Classification; Understanding the Failure Modes of Out-of-Distribution Generalization; relationship to calibration

Neural Collapse: the paper, a criticism

Double descent: the survey, exploration in deep learning, tons of other followups

Relatedly, interpolation learning: follow citations from e.g. this paper; Does Learning Require Memorization?;

An Equivalence Between Private Classification and Online Prediction, but A Computational Separation between Private Learning and Online Learning

Fancy concentration inequalities for learning bounds

Research idea: Non-vacuous bounds and testing: try using various non-vacuous generalization bounds in a classifier two-sample test. Could definitely turn into a nice workshop paper / potentially a full paper with some effort, but can do a small version for the project; talk to me if you're interested.