Mark Schmidt's Home Page

Mark Schmidt
    Department of Computer Science
    University of British Columbia
    201 2366 Main Mall
    Vancouver BC V6T 1Z4
    Canada

2017-Present: Alfred P. Sloan Research Fellow.
2017-Present: Canadian Institute for Advanced Research (CIFAR) Senior Fellow, Learning in Machines and Brains
2016-Present: Canada Research Chair in Large-Scale Machine Learning.
2014-Present: Assistant Professor (University of British Columbia), Laboratory for Computational Intelligence.
2013-2014: Postdoc (Simon Fraser University), Natural Language Lab
2011-2013: Postdoc (Ecole Normale Superieure), INRIA SIERRA project.
2010: Postdoc (University of British Columbia), Scientific Computing Lab
2005-2010: Ph.D. student (University of British Columbia), Laboratory for Computational Intelligence.
2006: Intern (Siemens Medical Solutions), Computer-Assisted Diagnosis and Therapy Group.
2003-2005: M.Sc. student (University of Alberta), Alberta Ingenuity Center for Machine Learning.

Research

List of Publications - with links to code, presentations/posters, and appendices.

Selected recent papers on optimization for machine learning:

arXiv 2018 (Fast and Faster Convergence of SGD for Over-Parameterized Models (and an Accelerated Perceptron))
OPTL 2018 ("Active-set complexity" of proximal-gradient: How long does it take to find the sparsity pattern?)
arXiv 2017 (Let's Make Block Coordinate Descent Go Fast: Faster Greedy Rules, Message-Passing, Active-Set Complexity, and Superlinear Convergence)
ECML 2016 (Linear Convergence of Gradient and Proximal-Gradient Methods under the Polyak-Lojasiewicz Condition)
NIPS 2015 (Stop Wasting My Gradients: Practical SVRG)
ICML 2015 (Coordinate descent converges faster with the Gauss-Southwell rule than random selection)
AI/Stats 2015 (Non-Uniform Stochastic Average Gradient for Training Conditional Random Fields)
arXiv 2013 (MAPR 2017) (Minimizing Finite Sums with the Stochastic Average Gradient)
ICML 2013 (Block-coordinate Frank-Wolfe Optimization for Structural SVMs)
NIPS 2012 (A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets)

Selected computer vision applications:

Other selected applications:

PhD thesis (2010): Graphical Model Structure Learning with L1-Regularization

Presentations

Talk slides on selected recent projects:

NIPS OptOpt 2016 (Do we need "Harmless" Bayesian Optimization and "First-Order" Bayesian Optimization)
ECML 2016 (Linear Convergence under the Polyak-Lojasiewicz Inequality)
ISMP 2015 (Advances in the Minimization of Finite Sums)
Opt and Big Data 2015 (Is Greedy Coordinate Descent a Terrible Algorithm?)
NIPS OPT 2014 (Non-Uniform Stochastic Average Gradient for Training Conditional Random Fields)
Google 2014 (Minimizing Finite Sums with the SAG Algorithm)
ICML 2013 (Block-Coordinate Frank-Wolfe Optimization for Structural SVMs)
NIPS 2012 (A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets)
NIPS 2011 (Convergence Rates of Inexact Proximal-Gradient Methods)
Xerox 2011 (Hybrid Deterministic-Stochastic Methods for Data Fitting)

Overview talk slides:

SIOPT 2017 (Converging on the Ultimate Algorithm for Minimizing Convex Sums)
Northwestern 2016 (The Why and How of Releasing Applied Math Code)
TAAI 2014 (Tractable Big Data and Big Models in Machine Learning)
Kyoto 2014 (Opening up the black box)
Paris-6 2013 (Linearly-Convergent Stochastic-Gradient Methods)
Edinburgh 2012 (Inexact Proximal-Gradient Methods and Linearly-Convergent Stochastic-Gradient Methods)
INRA 2011 (Structure Learning in Undirected Graphical Models)
NIPS OPT 2010 (Limited-Memory Quasi-Newton and Hessian-Free Newton Methods)
Thesis Talk 2010 (Graphical Model Structure Learning with L1-Regularization)

Selected recent posters:

NIPS BayesOpt 2016 (Do we need "Harmless" Bayesian Optimization and "First-Order" Bayesian Optimization?)
ECML 2016 (Linear Convergence of Gradient and Proximal-Gradient Methods under the Polyak-Lojasiewicz Condition)
UAI 2016 (Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions)
UAI 2016 (Convergence Rates for Greedy Kaczmarz Algorithms and Faster Randomized Kaczmarz Rules Using the Orthogonality Graph)
NIPS 2015 (Stop Wasting My Gradients: Practical SVRG)
ICML 2015 (Coordinate descent converges faster with the Gauss-Southwell rule than random selection)
AI/Stats 2015 (Non-Uniform Stochastic Average Gradient Method for Training Conditional Random Fields)
ICML 2013 (Block-Coordinate Frank-Wolfe for Structural SVMs)
NIPS 2012 (A Stochastic Gradient Method with an Exponential Convergence Rate for Strongly-Convex Optimization)
NIPS 2011 (Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization)

Recent videos of talks by myself and co-authors:

ICML 2015 (Coordinate Descent Converges Faster with the Gauss-Southwell Rule than Random Selection)
NIPS OPT 2014 (Non-Uniform Stochastic Average Gradient for Training Conditional Random Fields)
BIRS 2014 (Minimizing Finite Sums with the Stochastic Average Gradient)
Boulder 2014 (Fast Non-Smooth and Big-Data Optimization)
ICML 2013 (Block-Coordinate Frank-Wolfe Optimization)
NIPS 2012 (A Stochastic Gradient Method with an Exponential Convergence Rate)
NIPS 2011 (Convergence Rates of Inexact Proximal-Gradient Methods)
NIPS OPT 2010 (Limited-Memory Quasi-Newton and Hessian-Free Newton Methods)
AI/Stats 2010 (Convex Structure Learning in Log-Linear Models: Beyond Pairwise Potentials)
AI/Stats 2010 (Modeling annontator expertise: Learning when everybody knows a bit of something)

Code

List of Software Packages

Some Highlights:

matLearn (2016, machine learning in Matlab)
SAG4CRF (2015, stochastic average gradient for conditional random fields)
SAG (2013, L2-regularized logistic regression)
thesis (2012, code from my thesis)
- examples with simple regularizers
batching (2011) (growing-batch optimization)
UGM (updated 2011) (undirected graphical models)
L1General (updated 2010, L1-regularized optimization)
- examples
minConf (2009, optimization with simple constraints)
- examples with bound constraints
- examples with simple constraints
lasso (updated 2008, L1-regularized least squares)
minFunc (updated 2012, differentiable optimization)
- examples

A package containing most of the above is available here.

Teaching and Research Groups

80 Lectures on Machine Learning - material from all my courses in one place.

UBC Courses:

CPSC 340 and 532M: Machine Learning and Data Mining (Fall 2018, Fall 2017, Fall 2016, Fall 2015).
CPSC 540: Machine Learning (Winter 2019, Winter 2018, Winter 2017, Winter 2016, Fall 2014).

Mini-Courses:

SVAN 2016 (Stochastic Convex Optimization Methods in Machine Learning - videos)
MLSS 2011 (Convex Optimization)

Tutorials:

SIOPT 2017 (Stochastic Optimization for Machine Learning: Weakening the Assumptions)
DLSS 2015 (Part 2: Non-Smooth, Non-Finite, and Non-Convex Optimization - video)
DLSS 2015 (Part 1: Smooth, Finite, and Convex Optimization - video)
ICML 2015 (Modern Convex Optimization Methods for Large-Scale Empirical Risk Minimization - video)
ICASSP 2015 (Convex Optimization for Big Data)
MLSS 2015 (Convex Optimization - video)
ACML 2014 (Convex Optimization for Big Data)
UBC 2009 (Linear Algebra)

UBC Machine Learning Lab:

Current members:

Mohamed Ahmed (Ph.D.)
Reza Babanezhad (Ph.D.)
Mehrdad Ghadiri (M.Sc.)
Issam Laradji (Ph.D.)
Wu Lin (Ph.D.)
Michael Przystupa (M.Sc.)
Yasha Pushak (Ph.D.)
Alireza Shafaei (Ph.D.)
Sharan Vaswani (Ph.D.)
Nasim Zolaktaf (Ph.D.)

Alumni:

Saeid Allahdadian (Postdoc, now at Oracle)
Ricky Chen (M.Sc., now Ph.D. at Toronto)
Hamed Karimi (Postdoc, now at Q.I. Leap)
Raunak Kumar (B.Sc., now Ph.D. at Cornell)
Julie Nutini (Ph.D., now at UrtheCast)
Geoff Roeder (B.Sc., now Ph.D. at Princeton)
Jennifer (Xin Bei) She (B.Sc., now M.Sc. at Stanford)
Behrooz Sepehry (M.Sc., now at 1QBit)
Alim Virani (B.Sc., now at Google)

UBC Machine Learning Reading Group (ML Theory Reading Group)

Selected Notes

List of Notes

Cauchy's original paper from 1847 on gradient descent is available here.

I've also been known to swim, spike, and dunk.