Policies - Tamara Munzner Last revision: Tue Jan 30 15:05:10 2024 ------------------------------------------------------------------------ Paper and Talk Writing ------------------------------------------------------------------------ - We aim for a go/no-go decision three months before the paper is due. That's typically the time when the main work should be complete (code complete for technique papers, near-final prototype deployed to target users for design studies, experiment run for evaluation papers). There may still be some analysis to do, like computational benchmarks or parameter tuning, but by this milestone you should be turning most of your energy into writing it up. Most first-time authors are surprised by how long it takes to do this phase well. This timeline might sound insanely long to you at first glance. But explaining your ideas clearly in writing in fact takes an immense amount of time. The process I describe below is designed to achieve high-quality papers; our typical hit rate is far better than the global acceptance rate of 25%. If your project hasn't hit this target by then, we'll probably decide to slip to a later deadline (the next reasonable conference, or direct to journal, or whatever). Sprinting for an unlikely-to-be-reached target is a waste of everybody's time and energy - yours and mine. I'm not a fan of inflicting half-baked stuff on reviewers, it wastes their time and doesn't do great things for your reputation. I'm also not a fan of the "Least Publishable Unit". We aim for one excellent paper that achieves high impact in a good venue, when the work is solid and ready to go, rather than trickling out several papers at lower-tier venues that are partially redundant with each other. - Before you start the writing phase for a paper, you will give a "pre-paper" talk. That's the talk almost as if you're presenting the work at a conference, although it should mirror the paper structure. You won't start writing prose until we're happy with the pre-paper talk, which becomes the outline for the paper. It's OK for some parts of the results section to still be blank, with a title like "Fabulous Result Here", if that work is still going on in parallel with this writeup process. This approach helps you distill the critical bits, rather than getting caught up in writing individual words: it's much, much easier to iterate talk slides than prose. Crucial goals are to articulate the contributions (which always diverge from what we originally had in mind at the start), figure out what high-level exposition is required to make the core ideas understandable by readers who don't have all the context that's in your own head, get the order of the exposition right. It's an iterative process, the high-level framing and spin depend on the contributions, but it's hard to figure those out without the skeleton of the paper. This process is like rapid prototyping for paper writing, the result is a much better paper than if you only have time for a first draft of all of these things. You're also pushed to create key diagrams early, which will typically be used in the paper itself. We will go through multiple iterations of the pre-paper talk slides in our one-on-one meetings. When it's solid enough, you will present it at a group meeting. Plan on 30-40 minutes of talking and an hour to get feedback. We will inevitably discover places to make it better, including clarification of the contributions and reordering of exposition. You'll then iterate on the deck until we're both satisfied. Then, and only then, will you start writing the paper itself. We aim for the pre-paper talk delivered to the group around two months before the paper deadline, with some load balancing to spread things out. That schedule means you'll need to start on these slides multiple weeks before that time. If you're starting from scratch and it's your first time, best to plan on a full month. If you've been gradually iterating on and fleshing out slide decks throughout your project ("slide decks as a research method"), then it could take much less time because you've already got a solid base. - We will schedule a time for the group to read a reasonably complete paper draft one month before the deadline. Again, we'll jitter to spread things out during major paper writing seasons (Feb/Mar for VIS, Nov for EuroVis) if there are multiple papers in flight. By that point, we should have done multiple rounds of iteration. Typically you will write a first draft, I'll do a reading pass and we'll discuss how you should make changes to a second draft, and it will be time for me to take the writing token for a full pass after that, then you'll have the writing token back to do some smaller stuff. All of these phases should happen before the group reads it. Part of the reason I have this process in place is to spread things out so that I only have the writing token on one paper at a time. After the group's feedback (inevitably there's substantial and useful comments despite all the work we put into the pre-paper talk phase), we'll keep iterating on the paper. In a perfect world we've got things very solid multiple days before the deadline and we're down to final proofreading by the day before. - I will provide a lot of help with your first paper/talk, including doing quite a bit of actual writing as needed. The model is that we pass the paper back and forth, gradually converging towards a final version. Typically you do a few passes, then I do a big pass, then you rewrite a few passes, then I do a smaller pass, until we're just checking for typos. If you're not a native speaker of English and your grammar isn't perfect then typically I'll do the very last proofreading pass to make sure no errors snuck in. As your graduate career continues, you need to take on more and more of the paper writing job yourself. For MS students who only do one paper, there won't be much of a progression. But by the time you're on your last paper as a PhD student, you should be a very substantial part of the writing, with structural guidance rather than heavy lifting from me. The model we want by the end of a PhD is you write a draft, I critique, you rewrite. Note that this model typically requires more time than "you write, then I rewrite". I will tend to write papers faster than many of you because I've done it a lot more. But your job is to learn how to do it yourself. In some UBC groups, the model is that for the final PhD defense talk you're explicitly not given *any* critique/feedback from your supervisor, although you can use any and all other sources for feedback including your fellow students. I don't go this far. Although my goal is to have already taught you to create a good paper talk by the end of a PhD, it's a whole level of abstraction higher to do a good talk about an entire PhD's worth of work. So I will still be in the loop for your defense practice talks. Ditto for your job talk, if you're going on the academic job market. That's a whole different level of skill. (Warning - it will take much longer than you think to create a great job talk, even if you've got a great defense talk!) - Do your literature search before starting the project. Take detailed notes, you'll want them later when you write up. See below for more. - The pre-paper talk will serve as an outline that you can use as the guide to write prose. Nevertheless, if you have a tendency to get stuck in wordsmithing, consider a sweepline approach, where you do multiple passes to fill in levels of detail. Start with section titles. For your next pass, fill in subsection titles. For your next pass, have one bullet point per paragraph. (Your pre-paper talk should get you to this point with almost zero additional thinking.) For your final pass, have a few bullet points per paragraph of the main points you want to make. Key process point: monotonically advance through the paper, do NOT go back and fiddle with what came before until the next pass. Only after you have these paragraph-level bullet points should you start to write in complete sentences. - Writing up is typically a combination of bottom-up (work up from results) and top-down (work down from master plan). Consider starting writing the paper shell (intro, previous work, algorithm) even before you have final results. A more extreme approach that's worth trying is to do this before you've started programming *anything*. Then we can decide if it sounds like a strong enough project to bother doing it! It's also a lot easier for us to discuss the algorithm you have in mind if you've written it down, and we may be able to improve it before you spend a long time on coding. It's almost always true that once you have the results, you will drastically change the spin of those early sections. But you'll probably end up with a better paper if you do it this way, and maybe even spend less total time. - If a paper doesn't meet my quality bar, it doesn't go out. As an author, you also get veto power over whether a paper is good enough to submit. - No matter how much experience you have with public speaking, you need to do practice talks. The first time you give a particular talk is always less coherent than later ones. So it should be in front of a few of your peers, not in front of the actual audience! Then you can refine it, both at the structural level and the slide-by-slide tweak level. This rule is true for me too, so I schedule myself for practice talks as well when I'm putting together a new thing. If you are not totally comfortable in front of an audience, then the need for practice is even more urgent. - At a lower level, see writing style discussion: https://www.cs.ubc.ca/~tmm/writing.html - At a higher level, see the Ten Simple Rules for Structuring Papers (Kording and Mensh, PLOS Computational Biology), https://www.biorxiv.org/content/early/2017/05/23/088278 Some of the advice doesn't exactly map on to the structure of a visualization paper rather than a biology/bioinformatics paper, but much of it is relevant. ------------------------------------------------------------------------ Work Time and Patterns ------------------------------------------------------------------------ - You are judged in the real world by results not effort. That's how I judge as well. If you deliver, then I don't care how much, where, or when you work, as long as you've got a reasonable amount of time/space overlap in the building during some standard hours to interact with me and the rest of the group. That is, be around and available for one-on-one meetings, for group meetings, and for discussions with your fellow students. This flexibility on how, when, and where you work is part of the joy of grad school. - However, if results don't seem to be flowing smoothly, then I'll inquire more and more closely about your working habits and possibly introduce explicit expectations on when you're here to help you get back on track. That's how grad school finishes in a finite amount of time. In some cases I will ask you to write down short reports on what you've been up to since our last meeting *before* our scheduled meeting time. In most cases this will a half-page or less, in bullet points. (Sometimes it will make sense to write more down, if there's some sticky algorithmic issue that's worth documenting for the record.) These writeups will help me, and your co-supervisor if you have one, understand what you're doing and how your time is being spent. They will also help you when it comes time to write up papers or your thesis. They may also help you understand how your spend your time, which can be surprisingly surprising! - I do not expect deathmarches on projects/papers. It's not an impressive sign of macho committment, it's a sign of poor planning. If you're consistently doing them, then you have a time management problem and/or an unrealistic expectations problem. I can't sign up to do them with you, I have too many students and too many papers each year for that to be wise on my part. I may occasionally be able to parachute in to help at the last minute, but you absolutely can't count on it. When you don't get enough sleep, you're physically damaging yourself, you're emotionally more brittle, and you're not as creative. Worse yet, you're not nearly as smart: the more sleep-deprived you are, the stupider you are. ALL OF THIS IS TRUE FOR ME TOO. I am trying hard to change my own work patterns to not do this at all, for papers or anything else. - I do expect you to work hard. Research is a game where you're directly competing with lots of other people who are just as smart as you are, and many of them are working very hard. It's also true that to a first approximation, the harder you work the sooner you will finish grad school. But I don't expect you to work all the time. If you don't have a life outside of work, you will eventually burn out and/or be deeply unhappy. The only question is whether it will happen during grad school or after grad school. Neither is good. There will be times when you're in sprint mode to meet a deadline. That's hard to avoid, and not such a bad thing. But you shouldn't be in sprint mode permanently. - You should take some vacation. A good rule of thumb is 3-4 weeks a year. You should let me know in advance when you'll be gone. If you're asking to be gone too much, or at bad times given previous committments like paper deadlines, I'll let you know. One of the benefits of academic life is tacking on vacation time at the end (or start) of a work trip to some interesting place. I fully expect you to take advantage of that perq! ------------------------------------------------------------------------ Meetings and Meeting Prep ------------------------------------------------------------------------ - We have infovis group meetings once a week. Sometimes there are more around crunch times, when lots of people have papers or talks to critique. When we're not in paper writing mode, we typically read other people's papers and critique the heck out of them, with lots of discussion of methods and meta-science and paper writing approaches in addition to the raw content. It's all preparation for critiquing the heck out of our own work. The typical format is we individually read the paper in advance, then we go around the room saying what we think at a high-ish level, and then if time do a second pass where we quibble about smaller details. I usually go last. The person who picks the paper for that week does *not* do any kind of formal prep like slides. The group meeting often includes several students that I don't directly supervise but have a strong interest in visualization, and have at least taken my grad course, and a few faculty (UBC or visiting researchers) who work in the area. Sometimes we do work-in-progress talks about our own ongoing work, or have an outside speaker. I also expect you to attend the larger MUX HCI reading group as the common case for people who are full time with me. If you're part-time remote (eg sitting in a co-supervisor's lab part of the time), we'll discuss what makes sense. I also typically expect my students to be part of the Designing for People (https://dfp.ubc.ca) cohort, which includes attending the monthly seminar during the whole time you're here, and doing the methods & interdisciplinary team project courses in your first year. - When I'm in town I have one-on-one meetings with each of you for at least an hour a week. In some terms I schedule 90 minute blocks to have some buffer time. Running late is perhaps one of my most irritating character flaws; although I aspire to self-improvement I'm not counting on it, so my current workaround is to schedule buffers. I typically cancel meetings when I'm travelling, although I'll arrange to Zoom for crucial stuff as needed. - Prepare for meetings with a brief agenda, due via email before the meeting. If there are links that would be useful (for example to a research log web page, or a video URL if we're skyping a remote collaborator) include those in the email too so I have them handy. Please include these links every time, even if it's the same link as you've sent me a zillion times before. Also, do whatever other preparation would be useful ahead of time. If it's a software demo, have that up and running on your machine before we start. If it would help to have screenshots, make them in advance - then they're useful both for our discussion and as archival material, see below. ------------------------------------------------------------------------ Research Logs and Archives ------------------------------------------------------------------------ - Different people have different styles of work and preferences about how to track it. You should develop a system that works well for you. Some people keep everything online, others use a mix of digital and paper. I've seen (and in some cases used personally) many different platforms or combinations thereof for logging your research including: research log as pages on a private wiki (password protected), research log as plain HTML pages constructed by hand, bound paper research notebooks, looseleaf paper later filed into folders, papers you read printed out as hardcopy and kept with scribbles in the margins, papers you read systematically downloaded into local files on disk (all of them or just high-priority ones), papers you read recorded into citation infrastructure systems like Mendeley or Zotero or bibtex, logging nothing and optimistically assuming you'll just remember everything. I do not recommend that last one! It's particularly useful to write down what you think about papers as you read them when you're doing an in-depth literature search, especially at the start of a project. Details that are sharp and clear at the moment you read will be fuzzy, and even the big picture of what something is about and how it fits into your own work may completely evaporate if you didn't make your own personal notes when it comes time to write up months or even years later. The best case is you redo work, the worst case is you forget something crucial and it comes back to bite you (eg uncited crucial related work that causes reviewers to reject your paper). Some people write down detailed summaries plus commentary (including relevance to their intended research project) as they go, on a research log, for all papers. Others at least track the stuff they run across that's relevant, in an explicit list. Others (I admit often including me) read broadly and rely on memory to suffice, which is much much riskier. There's a tradeoff between reading stuff and writing down what you think about it: the latter certainly does take time away from the former. Your personal sweet spot may change over time: it's typically a very good idea to do a substantial amount of writing down at the start of your grad studies, but maybe less so as time passes and you develop your own personal style of research. - If you have an online research log, it's usually a good idea to have something beyond just text, where you routinely and systematically post images or graphs - whether daily for yourself as you go, or in some sort of weekly writeup as preparation for our meeting. That's especially important when there would be value in side by side comparisons of snapshots from different versions of algorithms, or performance graphs, or stuff like that. We know from infovis dogma that side-by-side comparison almost always trumps remembering what you saw before, so most discussions that involve gesturing at the screen would benefit from this! (Exception is stuff about animation, I don't expect you to make movies at the drop of a hat since that's too time-consuming.) It's also useful to have screenshots because we may want to include them in supplemental materials, showing for example the evolution of the interface through iterative cycles. These are typically trivial to record at the time but can be remarkably painful to recreate later. Your future self will benefit if your current self does it as you go! For your research web page, one very easy way to go is a simple log format with old on the bottom and new stuff added to the top. When you add something, do include a timestamp. If the page gets so full that it's unwieldy, you can start a new one and include a link to the old. Security through obscurity is fine - just don't link to this private research page from your public web page. If you want to do more sophisticated cross-referencing, then a wiki could be a good way to go. ------------------------------------------------------------------------ Version Control and Archiving ------------------------------------------------------------------------ - Use real version control for any and all source code that you write that is work I'm paying you to do. It's up to you whether to do it for personal projects, but I highly highly recommend it. Don't just make occasional tarballs, and/or comment old stuff out. Disaster will inevitably strike. You will need to be able to recover your code to the exact state that it was at some previous time. Sometimes it's because you went down a coding rathole and just want to get out. Sometimes it's because we need to compare a later algorithm to a previous algorithm, whether in a discussion or for a paper results section. The obvious choice these days is git as tooling and github for repos. (In previous decades it was svn/subversion for tooling and sourceforge for repositories. Before that was cvs and rcs, but now I'm dating myself.) - Use reasonable naming conventions for data filenames right from the start. Don't save it as "foo" and plan to rename it later. Disaster, where you lose work that could take from minutes to days to weeks to regenerate, will inevitably strike. If you're changing some parameters, try to encode them in the filename. If that gets unwieldy, record filenames and parameters in a README file that you keep in the directory. Don't forget that if your code is changing a lot, it probably doesn't make sense to always overwrite the same file, even if your parameters don't change. Consider making dated directories or something, so that old data files are still accessible instead of blown away. Everything I said about code above is true for data - we will often need older stuff for algorithm comparison or paper figures. There's now a great slide deck from Jenny Bryan about Naming Things. Read it and follow her recommendations! https://speakerdeck.com/jennybc/how-to-name-files - Back up your work. If you keep all your stuff on the main UBC CS filesystem, that's automatic. If you do any work on a laptop, you need to set up something explicitly, like Dropbox or Google Drive in the cloud or some kind of private hardware setup of your own, and do it regularly. Do not put this off, disaster will inevitably strike. On the personal data front - don't forget about backing up your phone too! - Archive your (non-scratch) work in a way that I can have access to it later at /ubc/cs/research/imager/project/infovis/. That includes research papers (source and final), talk slides (all versions not just final, sometimes slides we cut along the way are useful in other contexts), official docs like thesis proposals and annual reports, source materials like interview transcripts or benchmark datasets. Have README files explaining what's what if it's not obvious, especially for example provenance of datasets. It's best if you keep this archive directory updated as you go. I'll definitely ask you to confirm it's all done before I sign off on your request-to-graduate forms. - These days for filesharing and collaboration and paper writing we use a ridiculously ad hoc mix of Dropbox, GoogleDrive, OneDrive, Overleaf (which I sync with Dropbox and could also sync to github), and sometimes the UBC CS filesystem. I am not thrilled with this unholy mix, making it hard to find things, but I haven't figured out a standard way that's better than ad hoc chaos. (I live in hope...) For paper writing, if at all possible we will use LaTeX. I'll make exceptions only if there's a senior collaborator who is invested in Word and it's not realistic for them to learn LaTeX. I am strongly against emailing files back and forth, that's a desperation measure. Dropbox or GoogleDrive, where we use filename conventions if we're actively collaborating on the same document, is one way to go. For papers, Overleaf has the crucial benefit of making diffs easy to browse. I pay for the professional version of Overleaf, so have me create the initial project that you join - that way we have version history. I use graphical diff extensively during our back-and-forth writing passes, it's absolutely crucial that I'm able to see the differences from the last version I saw and the current one. One problem is that it's a pain to recover old versions of files - so explicitly save off versions at useful milestones. Also use labels to make those easy to find. Do NOT use the Overleaf comments feature, I won't be able to find things if I'm looking at the file offline using a normal text editor. Use comments within the LaTeX source. Within Overleaf, please archive the PDF versions of crucial milestones (submit, camera-ready) for main paper and supplemental material, and also for response to reviewer for camera-ready. Archive the raw reviews in a text file as well. GoogleDocs are great for realtime collaboration, and for browsing changes via graphical diff, but terrible for archiving: what's stored locally on your GoogleDrive folder is just a URL, not its contents. I don't have a good solution at this point, but I'm frequently frustrated. Make a folder within GoogleDocs (and/or OneDrive) where you keep all the files we need to have shared access to, and share that with me with permissions set properly, and use subfolders within that as needed. Make sure that all new gdoc/gsheet/gdrive files live in that space. ------------------------------------------------------------------------ Reproducible Research ------------------------------------------------------------------------ - Reduce pain with preplanning. Everything Sylvain says is spot on! http://sylefeb.blogspot.com/2011/09/research-producing-results.html - In a perfect world we should be able to (painlessly) reproduce every figure that goes into a paper. Think about developing a workflow that would support that goal. Your future self will be happy, and so will I. ------------------------------------------------------------------------ Supervision ------------------------------------------------------------------------ - Taking my graduate course, and doing well, is a prerequisite for being my student. I have structured my course so that there's a high correlation between doing well in it and being a good match for working with me. (It's not a one-way function, of course; many people do well who are not intending to work with me!) Moreover, the course will give us both a chance to see if we're a match in terms of personality and research style. At one point my process was to decide in advance how many open slots I have, and then go through the ranked list of students after the course ends and offer positions in order starting from the top. We now have a somewhat different process where PhD track and PhD student admissions involve commitments of support in advance. If I have previously and explicitly committed to working with you as a PhD student, I will typically prioritize that agreement - but even many PhD students come in with an understanding that you might work with one of a small set of co-sponsoring professors. For MSc students it's very common to shift to a different supervisor than you originally had in mind, part of the role of our graduate courses is to help you shop around to see what resonates. I've had some people decide to work with me who didn't originally intend to do visualization, but they got hooked through taking my course. In all of these cases, taking my course is an important part of us both understanding whether there's in fact a good supervisory match. I will make exceptions for years I'm not teaching my grad class (such as when I'm on sabbatical), we'll discuss on a case by case basis. - I strongly value communication skills, both writing papers and giving talks. I will work with you to improve your skills over time, using the quite intensive processes that I describe in this document. Many non-native speakers of English have been extremely successful in my group, but you do need a high enough level of English proficiency at the start of your degree program that we can communicate clearly and fluidly. It's crucial that we can have in-depth discussions without a language barrier. However, I am not looking for grammar perfection. I have worked with many great people who never could write perfectly mistake-free English even after many years of living in an English-speaking country and/or working with me - but they became great writers in terms of high-level structure and mid-level flow. I'm fine on doing grammar clean-up passes as we go and at the end, for me reading and copyediting quickly comes easily. - The departmental RPE (Research Proficiency Exam) process is primarily designed to help us both figure out quickly if you're likely to succeed as a researcher, and secondarily intended to let us figure out if we work well together. I will work with you to design an RPE project that's intended to be a hunk of work towards the first paper that you'll write, so that you're making progress towards publication rather than doing something that's ultimately a side track. It is NOT intended to result a complete paper; the paper writing process I describe below is heavy-weight enough that it can't fit within the RPE time frame, plus it's highly unlikely that a full paper's worth of work could be done within that window even if no writing time is required. If you've done a successful MSc with me, then you're not required to do an RPE as part of your PhD program. We should still have an explicit checkin after around six months, to see if both you and I feel that the collaboration is working well. If so, then I commit to being your supervisor for the rest of your PhD (assuming reasonable progress towards a degree). If not, then we part ways and you will need to find another supervisor. If you're doing a research MSc, then the RPE process does not apply. If you're a PhD track master's student, concluding the RPE is the point at which you transfer from the MSc program to the PhD program (and your stipend also increases accordingly).