Topics in Computer Graphics / AI (CPSC 533R)

Deep Learning for Computer Vision and Graphics

Winter Term 1, 2020/2021 - Course Information

Computer Graphics (CG) enables artists to realize their creative visions. Technically, one is concerned with efficiently simulating light transport for photorealistic rendering, finding the right parametrization for the shape and appearance of objects, and making these digital models accessible to the artist. Creating digital content is a tedious process, which is alleviated by Computer Vision (CV) by reconstructing real scenes from natural images.

This course will explore the recent trend of using Deep Learning (DL) for the above-mentioned tasks; neural rendering, generative networks, disentangled representation learning, and reconstruction using neural networks. The part of the course will introduce the essential tools (CNN architectures, attention models, output representations, and batch-norm layers), those needed to understand the advanced topics covered in the main part (self-supervised and generative models, such as VAEs and GANs).

A central theme is the learning of neural representations from labeled and unlabeled data. At one end, we will discuss approaches that can reconstruct a person photorealistic, from a single image and up to the scale of wrinkles, by using a volumetric grid with thousands of parameters. On the other end, we will study techniques for learning compact and meaningful embeddings, such as a parametric face model that decomposes expression, appearance, and pose and makes these latent dimensions controllable by the artist. At the end of the course, students will be able to work with a diverse spectrum of DL tools and be prepared to conduct research on how to create deep learning models that yield both, high visual quality and have artistic utility.

Specials for running online (new in 2020 Winter Term 1)

Studying in isolation at home sounds boring. To counteract, every session will have interactive elements, such as 3 minutes group discussions in breakout rooms and jointly working on preliminaries for the course assignments. Moreover, assignments are now solved in groups of two (instead of individually), with changing teammates to imitate incidental conversations and cooperations that happen naturally in a physical classroom.

All classes will be live on Zoom via the Canvas platform (see instructions below), which enables students to post questions and interact with each other with video, audio, and text chat. Lecture slides and jupyter notebooks will be made available, yet class attendance is required. For remote students for whom the live lectures are at night, we do a replay session later the same day (see times below). The replay session will be in the same zoom room as the lecture (you can access it via Canvas). On top of the classes, office hours will be offered by the TA and me (instructor) once a week; time TBD in the first week, based on your constraints. Moreover, both of us will be available for immediate discussion and questions on the Piazza discussion platform.

Do I need to prepare for online courses? Yes, make sure to have a reliable high-speed internet connection, a device capable of running high-resolution live streams, and a good headset. Please contact the instructor, TA, or UBC-CS support team should you have trouble with either of these, we will try to support. The headset might be most important, to be able to discuss teamwork and participate in the lecture with high sound quality (mics integrated into laptops have poor quality) and without triggering a feedback loop (running loudspeakers and mic is bound to do so). We also recommend setting up your home office with a comfortable chair, a big screen, and a webcam facing a blank background to maintain privacy (e.g., facing a wall, curtain, or professional separator as streamers use). More suggestions on https://keeplearning.ubc.ca/setting-up/#technical.

Disclaimer. During this pandemic, the shift to online learning has greatly altered teaching and studying at UBC, including changes to health and safety considerations. Keep in mind that some UBC courses might cover topics that are censored or considered illegal by non-Canadian governments. This may include, but is not limited to, human rights, representative government, defamation, obscenity, gender or sexuality, and historical or current geopolitical controversies. If you are a student living abroad, you will be subject to the laws of your local jurisdiction, and your local authorities might limit your access to course material or take punitive action against you. UBC is strongly committed to academic freedom, but has no control over foreign authorities (please visit http://www.calendar.ubc.ca/vancouver/index.cfm?tree=3,33,86,0 for an articulation of the values of the University conveyed in the Senate Statement on Academic Freedom). Thus, we recognize that students will have legitimate reason to exercise caution in studying certain subjects. If you have concerns regarding your personal situation, consider postponing taking a course with manifest risks, until you are back on campus or reach out to your academic advisor to find substitute courses. For further information and support, please visit: http://academic.ubc.ca/support-resources/freedom-expression

How to access Zoom? You can access the Zoom room by selecting the course CPSC 533R in Canvas -> Zoom at the top on the left menue -> select any leccture -> click on Join URL link.
We encourage you to join with video, but this is up to your discretion and privacy. There is a 'raise hand' button on the bottom of the screen, use that to indicate as you would do in a real class. There is also the option to join by phone, for those that have an unlreliable internet connection. Please be in time (the room opens is open in advance!) for the class to not making the others wait. Come some minutes early and chat with whoever is already around, as you would do in a physical classroom.

Course details

Registration and official listing: CPSC 533R

Instructor: Helge Rhodin (rhodin@cs.ubc.ca)
Office hours (drop-in): Tuesday 5:30 PDT and Thursday 10:30 PDT, on Zoom

Teaching Assistant: Yuchi Zhang (tentative)
Office hours (drop-in): Friday 3 pm, on Zoom

Classes: Tuesday and Thursday 11:00 12:30 PDT (winter term 1)
Replay session: 6 pm PDT on all lecture days
Room: online, Zoom (see instructions above)

Piazza: Discussions will be on Piazza.com/ubc.ca/winterterm12020/cpsc533R

Compute: The assignments and course projects will deal with images and require CUDA-capable GPUs for training. You can either use your own GPU, open cloud services like google colab, or UBC machines lin01.students.cs.ubc.ca to lin25.students.cs.ubc.ca.

Lectures (preliminary schedule)

The lecture schedule will be updated with links to lectures, assignments, and student presentation slots.

Prerequisites

Machine Learning. It is required to first study CPSC 340 or an equivalent course. Before starting this course, the following ML terms should be familiar to you: stochastic gradient descent and momentum; classification and regression; weight initialization and data whitening; training, validation and test set; convolution and translational invariance; and regularization and weight decay. Please contact me if you have any doubts.

Python. The exercises are designed to prepare students for the practical project and provide a step-by-step introduction to the PyTorch machine learning framework. While no prior knowledge on PyTorch is required, essential python experience is expected; no python programming tutorial will be offered.

Prior courses in CG and CV are a plus and course projects that build upon these are encouraged.

Grading

Assignments (25%): Presenting research (35%): Project (40%)

Auditing

Audits are permitted on a case-by-case basis; please contact me prior to the start and join for the first lecture. Auditing students have to participate actively in the reading part of the class, i.e., reading of the papers, writing a review, and participating in the discussion. Audits are more than welcome to also work on the assignments or course project, but this is not a requirement.

Assignments

The course contains three assignments that add points to the final grade, however, their primary goal is to prepare you for the project. The first assignment is to set up a general training framework that lets one experiment with different datasets, neural network architectures, and objective functions. This framework will be extended and used in the remaining two assignments and also the final course project. Instead of step-by-step instructions, exercises are posed as high-level tasks. Questions are set up to train independent thinking, which is integral to solving actual problems in research and industry settings. This does not mean that you are without support; you are highly encouraged to search online and consult the abundance of deep learning tutorials as a basis for your solution. Furthermore, office hours are set up to provide personal assistance.

Academic Integrity. Assignments must be solved in teams of two, using the available course material and other online sources. You are neither allowed to copy nor look at parts of any of the assignments from anyone else except your teammate. Accordingly, it is not allowed to post solutions online or distribute them in (private) forums. The university policies on academic integrity are rigorously applied. Teamwork and submission. One solution must be submitted per group via Canvas, but we expect that the solution was derived together (via video chat sessions) and both members must be able to explain all parts. Groups are assigned for each assignment at random by the instructor to facilitate diverse interactions and equal opportunities.

Deadline and grading. Assignments will be due on the dates specified in the schedule, always at 11:59 pm PST. A late submission by one day will still be accepted but reduces the score by -25%. The grading is based on correctness and completeness, as detailed in each exercise description.

Research papers

The second half of the course will constitute a reading group, focussing on seminal and latest research. Each student will write reviews every week and is expected to participate actively in the discussion, along with presenting once during the course. Preferences will be considered for the presentation topic assignment.

Reviews. All students are expected to read one of the two papers that will be presented and discussed in each class. Nevertheless, it is recommended to at least skim through both the papers. Each presentation will be followed by an in-depth discussion on technical aspects, relations to other works, and practical utility. To initiate and start new arguments, a short review of the selected paper will have to be submitted on the day before the class. Try to raise non-obvious aspects with the following:

Keep your review concise (less than half a page) and self-contained.

Moderation. Each presentation is followed by a ~15-minute discussion. You will moderate the discussion of the paper presented by somebody else in the lecture following yours. It is your task to start the discussion by asking suitable questions and leading the subsequent discussion. We will provide you with reviews of the other students, which you can use as a starting point. Note, the moderation replaces the peer-review done in previous iterations of this course.

Presentation. Every student presents one research paper. The presentation should be compact and precisely 15 minutes long. Focus on the most crucial points, those needed to understand the content, novelty, and impact of the given work. Most importantly, draw links to previous presentations, lecture content, and the related work you know. Such an embedding into a broader context will make your presentation unique and stimulating, especially for those who have read the underlying paper in detail. You can take the paper illustrations and other available online material, such as videos supplementing the paper, as a basis for your presentation, as long as you reference the source adequately on each slide. You will be the expert and be required to study the paper in sufficient depth to answer questions and trigger a rich discussion.

Preparing your presentation: You can either present live (more interactive but error-prone in case of unreliable internet connection/software failure) or pre-record the presentation as described in the following slides (recording instructions) and we will watch them together (you must be present) during the normal lecture time.

If you chose to record, you should record in a single run, as giving the presentation live. Export your recorded presentation as Powerpoint file (voice and slides) or as a video (e.g., mp4). Powerpoint and keynotes are easy to use. You can still develop slides in google slides and other tools, simply export them to the Powerpoint format.

If you haven't done such a recording before, try it out as soon as possible; at least a week before your presentation. Practice your talk before, ideally in front of some friends, to get the timing and the messages right. Note that a part of your audience will have read the paper in detail and others will only have skimmed through. Therefore, try to address experts and newcomers alike. The tutor will assist you in this and give feedback on your planned presentation, make an appointment with him in advance.



Deadlines and grading.

Project

The whole course is structured to prepare for the final research project. The goal is to either extend an existing approach, combine existing tools for solving a new problem, or to come up with an entirely new algorithm. Some topics and open research problems are given in class, but we highly encourage you to come up with your own idea on the basis of your unique background and experience. Projects can deviate from the core lecture topics, as longs as they are within CG, CV, and ML. You are encouraged to work in groups of two to three students. Joining forces will allow you to complement your strengths and work on more challenging and rewarding problems. Nevertheless, all team members must have unique responsibilities, which have to be explicitly listed, and be involved in all of the following project stages:

Deadlines and grading. Project proposal, pitch, report, and final presentations will be due on the dates and times specified in the schedule linked above (pitch ~mid-term, project presentations in the last two weeks, partial report submissions during the term, final report submission on the 14th of Dec.). A late submission reduces the gained points by -25% per day. The grade will mainly depend on the project definition, how well you present them in the report, how distinctly you position your work in the related literature, how thorough your experiments are, and how thoughtfully your contribution is derived. We value quality over quanity, focus on one aspect and keep things as simple as possible.