publications: journal and conference papers

Overview: The Design, Adoption, and Analysis of a Visual Document Mining Tool For Investigative Journalists

Matthew Brehmer, Stephen Ingram, Jonathan Stray, and Tamara Munzner
IEEE Trans. Visualization and Computer Graphics (TVCG, Proc. InfoVis), Volume 20, Number 12, 2014, p.2271-2280, (acceptance rate: 23%, 45/196).

For an investigative journalist, a large collection of documents obtained from a Freedom of Information Act request or a leak is both a blessing and a curse: such material may contain multiple newsworthy stories, but it can be difficult and time consuming to find relevant documents. Standard text search is useful, but even if the search target is known it may not be possible to formulate an effective query. In addition, summarization is an important non-search task. We present Overview, an application for the systematic analysis of large document collections based on document clustering, visualization, and tagging. This work contributes to the small set of design studies which evaluate a visualization system "in the wild", and we report on six case studies where Overview was voluntarily used by self-initiated journalists to produce published stories. We find that the frequently-used language of "exploring" a document collection is both too vague and too narrow to capture how journalists actually used our application. Our iterative process, including multiple rounds of deployment and observations of real world usage, led to a much more specific characterization of tasks. We analyze and justify the visual encoding and interaction techniques used in Overview's design with respect to our final task abstractions, and propose generalizable lessons for visualization design methodology.

→ [pdf] [overviewproject.org] [IEEE link] [project page]

C-TOC (Cognitive Testing on Computer): Investigating the Usability and Validity of a Novel Self-administered Cognitive Assessment Tool in Aging and Early Dementia.

Claudia Jacova, Joanna McGrenere, Hyunsoo S. Lee, William Wang, Sarah Le Huray, Emily F. Corenblith Matthew Brehmer, Charlotte Tang, Sherri Hayden, B. Lynn Beattie, and Ging-Yuek R. Hsiung
Alzheimer Disease and Associated Disorders, 2014.

INTRODUCTION:: Cognitive Testing on Computer (C-TOC) is a novel computer-based test battery developed to improve both usability and validity in the computerized assessment of cognitive function in older adults. METHODS:: C-TOC's usability was evaluated concurrently with its iterative development to version 4 in subjects with and without cognitive impairment, and health professional advisors representing different ethnocultural groups. C-TOC version 4 was then validated against neuropsychological tests (NPTs), and by comparing performance scores of subjects with normal cognition, Cognitive Impairment Not Dementia (CIND) and Alzheimer disease. C-TOC's language tests were validated in subjects with aphasic disorders. RESULTS:: The most important usability issue that emerged from consultations with 27 older adults and with 8 cultural advisors was the test-takers' understanding of the task, particularly executive function tasks. User interface features did not pose significant problems. C-TOC version 4 tests correlated with comparator NPT (r=0.4 to 0.7). C-TOC test scores were normal (n=16)>CIND (n=16)>Alzheimer disease (n=6). All normal/CIND NPT performance differences were detected on C-TOC. Low computer knowledge adversely affected test performance, particularly in CIND. C-TOC detected impairments in aphasic disorders (n=11). DISCUSSION:: In general, C-TOC had good validity in detecting cognitive impairment. Ensuring test-takers' understanding of the tasks, and considering their computer knowledge appear important steps towards C-TOC's implementation.

→ [publisher site]

A Multi-Level Typology of Abstract Visualization Tasks

Matthew Brehmer and Tamara Munzner
IEEE Trans. Visualization and Computer Graphics (TVCG, Proc. InfoVis), Volume 19, Number 12, 2013, p.2376–2385 (acceptance rate: 25%, 38/152).

The considerable previous work characterizing visualization usage has focused on low-level tasks or interactions and high-level tasks, leaving a gap between them that is not addressed. This gap leads to a lack of distinction between the ends and means of a task, limiting the potential for rigorous analysis. We contribute a multi-level typology of visualization tasks to address this gap, distinguishing why and how a visualization task is performed, as well as what the task inputs and outputs are. Our typology allows complex tasks to be expressed as sequences of interdependent simpler tasks, resulting in concise and flexible descriptions for tasks of varying complexity and scope. It provides abstract rather than domain-specific descriptions of tasks, so that useful comparisons can be made between visualization systems targeted at different application domains. This descriptive power supports a level of analysis required for the generation of new designs, by guiding the translation of domain-specific problems into abstract tasks, and for the qualitative evaluation of visualization usage. We demonstrate the benefits of our approach in a detailed case study, comparing task descriptions from our typology to those derived from related work. We also discuss the similarities and differences between our typology and over two dozen extant classification systems and theoretical frameworks from the literatures of visualization, human-computer interaction, information retrieval, communications, and cartography.

→ [pdf] [slides] [IEEE link] [project page]

Investigating Interruptions in the Context of Computerised Cognitive Testing for Older Adults

Matthew Brehmer, Joanna McGrenere, Charlotte Tang, and Claudia Jacova
Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI), 2012, p.2649-2658 (acceptance rate: 23%, 370/1577).

Interruptions in the home pose a threat to the validity of self-administered computerised cognitive testing. We report the findings of a laboratory experiment investigating the effects of increased interruption workload demand on older adults' computerised cognitive test performance. Related work has reported interruptions having a range of inhibitory and facilitatory effects on primary task performance. Cognitive ageing literature suggests that increased interruption workload demand should have greater detrimental effects on older adults' performance, when compared to younger adults. With 36 participants from 3 age groups (19-54, 55-69, 70+), we found divergent effects of increased interruption demand on two primary tasks. Results suggest that older and younger adults experience interruptions differently, but at no age is test performance compromised by demanding interruptions. This finding is reassuring with respect to the success of a self-administered computerised cognitive assessment test, and is likely to be useful for other applications used by older adults.

The Haptic Crayola Effect: Exploring the Role of Naming in Learning Haptic Stimuli

Inwook Hwang, Karon MacLean, Matthew Brehmer, Jeff Hendy, Andreas Sotirkopoulos, and Seungmoon Choi.
Proc. IEEE World Haptics Conference (WHC), 2011, p.385-390.

A haptic icon is a short physical stimulus attached to a simple meaning, which provides information and feedback to a user. To scale the utility demonstrated for small icon sets to larger ones, we need efficient strategies to help users learn subtle distinctions among stimuli, in a modality for which they may not hold detailed descriptive percepts. This paper investigates the effect of naming haptic stimuli – i.e. explicitly creating a linguistic marker – on the accuracy with which users are able to identify, distinguish, and recall stimuli.

We conducted a between-subjects experiment using 60 participants equally divided among three naming conditions: no names, pre-selected non-descriptive names, and self-selected names. The experiment examined the impact of naming strategy on the ability of participants to identify stimuli in a nonverbal matching test, and on remembering stimulus names. For this challenging task and the degree of learning afforded, naming did not significantly impact accuracy of matching stimuli to meanings for all participants. However, more than twice of many of those allowed to choose names reported the ability to remember and distinguish stimuli than those required to use non-descriptive names, and many participants felt that the names were useful. Of middle-performing participants, the self-selected names group performed significantly better than the non-descriptive names group, and appeared to progress more quickly in learning. We summarize evidence for a trend that might widen with refined naming strategies and more extensive learning.

Activate Your GAIM: A Toolkit for Input in Active Games

Matthew Brehmer, T.C. Nicholas Graham, and Tadeusz Stach
Proc. ACM Academic Conference on the Future of Game Design and Technology (Future Play), 2010, p.151-158.

Active games are video games that involve physical activity. nteraction in active games is captured via a variety of input devices such as accelerometers, cameras, pressure sensors and exercise equipment. It is difficult for programmers to deal with this profusion of devices, leading most active games to be tied to a particular hardware platform. In this paper, we introduce the GAIM toolkit. GAIM simplifies input handling in active games through a high-level API that abstracts the details of individual devices. This allows developers to write code independently of the input devices used, allows the toolkit to dynamically adapt to the devices a player has available, and allows people with different hardware to play together. We illustrate the approach through two active games developed using the GAIM toolkit.

Classifying Input for Active Games

Tadeusz Stach, T.C. Nicholas Graham, Matthew Brehmer, and Andreas Hollatz
Proc. ACM Advances in Computer Entertainment (ACE), 2009, p.379-382.

Active games are video games that involve physical activity. Active games capture input via a variety of devices such as accelerometers, cameras, pressure sensors and exercise equipment. Although active games have become highly popular, the interaction styles they support are poorly understood, and largely driven by the capabilities of individual hardware devices. In order to allow for a standard development approach, a better understanding of the interaction found in active games is required. We have investigated existing commercial and academic games in order to classify input for active games. Our classification abstracts input from hardware, providing a better understanding of the interaction itself. Our ultimate goal is to make it easier to develop active games independently of underlying input hardware.

workshop papers

Visualizing Dimensionally-Reduced Data: Interviews with Analysts and a Characterization of Task Sequences.

Matthew Brehmer, Michael Sedlmair, Stephen Ingram, and Tamara Munzner.
In Proceedings of the ACM Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV), p.1-8, 2014.

We characterize five task sequences related to visualizing dimensionally-reduced data, drawing from data collected from interviews with ten data analysts spanning six application domains, and from our understanding of the technique literature. Our characterization of visualization task sequences for dimensionally-reduced data fills a gap created by the abundance of proposed techniques and tools that combine high-dimensional data analysis, dimensionality reduction, and visualization, and is intended to be used in the design and evaluation of future techniques and tools. We discuss implications for the evaluation of existing work practices, for the design of controlled experiments, and for the analysis of post-deployment field observations.

Pre-Design Empiricism for Information Visualization: Scenarios, Methods, and Challenges.

Matthew Brehmer, Sheelagh Carpendale, Bongshin Lee, and Melanie Tory.
In Proceedings of the ACM Workshop on BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV), p. 147-151, 2014.

Empirical study can inform visualization design, both directly and indirectly. Pre-design empirical methods can be used to characterize work practices and their associated problems in a specific domain, directly motivating design choices during the subsequent development of a specific application or technique. They can also be used to understand how individuals, existing tools, data, and contextual factors interact, indirectly informing later research in our community. Contexts for empirical study vary and practitioners should carefully consider finding the most appropriate methods for any given situation. This paper discusses some of the challenges associated with conducting pre-design studies by way of four illustrative scenarios, highlighting methods as well as the challenges unique to the visualization domain. We encourage researchers and practitioners to conduct more pre- design empirical studies and describe in greater detail their use of empirical methods for informing design.

presentations / posters

Visualization Task Abstraction from Multiple Perspectives

Matthew Brehmer
IEEE VIS Docoral Colloquium, November 8th, 2014, Paris, France.

Applied information visualization design and evaluation involves a mapping between a domain problem and appropriate visual encoding and interaction techniques. This mapping translates a domain-specific workflow into abstract visualization tasks, which facilitates the comparison of visualization usage across application domains; the mapping also applies in reverse, whenever practitioners aim to contextualize novel visualization techniques. I propose four perspectives on visualization task abstraction: the synthesis perspective describes our recent typology of abstract visualization tasks. The remaining three perspectives involve validating this typology in applied settings: a post-deployment evaluation of visualization utility in the domain of investigative journalism, a visualization design study project in the domain of energy usage analysis, and a interview study spanning 11 scientific research domains pertaining to visualization tasks for dimensionally-reduced data.

→ [DC poster (pdf)] [Video Preview (2.6 MB mp4)]

Cognitive Testing on Computer (C-TOC): Development of a New Computer-Based Battery for Office and Home Administration

Claudia Jacova, Joanna McGrenere, Hyunsoo S. Lee, William Wang, Sarah Le Huray, Matthew Brehmer, Samantha Feldman, Charlotte Tang, Sherri Hayden, B. Lynn Beattie, and Ging-Yuek R. Hsiung
Poster at the Alzheimer's Association International Conference (AAIC), 2012.

Self-administered computerized test batteries offer an efficient and widely accessible alternative to current test approaches in facing the growing demand for evaluation of cognitive concerns. Our aim was to provide innovation in this area, with the design of Cognitive Testing On Computer C-TOC. Here we report on the design, iterative development and validation of this novel tool.

A Tale of Two Studies: Investigating the Impact of Interruptions on Task Performance in Older Adults

Matthew Brehmer, Charlotte Tang, Joanna McGrenere, and Claudia Jacova
Work-In-Progress Short Paper and Presentation at the GRAND NCE AGM, 2011.

We present two ongoing studies that are part of a larger multi-disciplinary research project which aims to design, develop, and implement a web-based computerized screening test for cognitive impairment, called Cognitive Testing on a Computer. A laboratory study is conducted to investigate the impact of interruptions on task performance among three different age groups. A qualitative field study is undertaken in older adults’ homes to examine potential interruptions and distractions that may be present as barriers to their performance when taking the assessment tool from home. Our initial findings indicate that technological interventions may help detect and mitigate interruptions for older adults.

technical reports

Dimensionality Reduction in the Wild: Gaps and Guidance.

Michael Sedlmair, Matthew Brehmer, Stephen Ingram, and Tamara Munzner
UBC CS Technical Report TR-2012-03, 2012.

Despite an abundance of technical literature on dimension reduction (DR), our understanding of how real data analysts are using DR techniques and what problems they face remains largely incomplete. In this paper, we contribute the first systematic and broad analysis of DR usage by a sample of real data analysts, along with their needs and problems. We present the results of a two-year qualitative research endeavor, in which we iteratively collected and analyzed a rich corpus of data in the spirit of grounded theory. We interviewed 24 data analysts from different domains and surveyed papers depicting applications of DR. The result is a descriptive taxonomy of DR usage, and concrete real-world usage examples summarized in terms of this taxonomy. We also identify seven gaps where user DR needs are unfulfilled by currently available techniques, and three mismatches where the users do not need offered techniques. At the heart of our taxonomy is a task classification that differentiates between abstract tasks related to point clusters and those related to dimensions. The taxonomy and usage examples are intended to provide a better descriptive understanding of real data analysts’ practices and needs with regards to DR. The gaps are intended as prescriptive pointers to future research directions, with the most important gaps being a lack of support for users without expertise in the mathematics of DR, and an absence of DR techniques for comparing explicit groups of dimensions or for relating non-linear embeddings to original dimensions.

theses

Usability and the Effects of Interruption in C-TOC: Self-Administered Cognitive Testing on a Computer

Matthew Brehmer
M.Sc. Thesis, Dept. of Computer Science, University of British Columbia, August 2011

Cognitive Testing on a Computer (C-TOC) is a self-administered web-based computerised cognitive assessment battery. C-TOC's intended scenario of use involves an older adult, who has presented a concern regarding his or her cognitive health, completing the test independently at home, as directed by their family physician or a specialty clinic.

This thesis presents the results of two studies aimed to address the viability of older adults completing the C-TOC test battery in a home setting, first to identify usability issues, and second to understand the effects of interruptions on C-TOC performance. In Study 1, an initial standard evaluation of C-TOC's usability was conducted with representative users and a cross-cultural advisory panel of health professionals. Based on our own observations of participants' interactions with C-TOC, together with subjective reporting measures (interviews, questionnaires, & focus group discussion), several user interface design issues were identified. Given these issues, this thesis presents a list of recommendations for improving C-TOC's usability in subsequent versions.

The bulk of the novel contributions presented in this thesis arise from Study 2. In this study, we report the findings of a laboratory experiment to investigate the effects of increasingly demanding interrupting tasks on older adults' C-TOC testing performance. Related work has reported interruptions having a range of inhibitory and facilitatory effects on primary task performance. Cognitive ageing literature has suggested that increased interruption workload demand should have greater detrimental effects on older adults' performance, when compared to younger adults.

With 36 participants from 3 age groups (19-54, 55-69, and 70+), we found divergent effects of increased interruption demand on two primary tasks. Results suggest that older and younger adults experience interruptions differently, that increased interruption demand can incur a task resumption cost. However, at no age is test performance, in terms of accuracy, compromised by demanding interruptions. This finding is reassuring with respect to the success of C-TOC, and is promising for other applications used by older adults.

It is our hope that what was learned from both studies will contribute to the development of a usable and valid cognitive assessment test.

Assessing the Effect of Exercise Intensity on Cognitive Task Performance in an Exercise Video Game

Matthew Brehmer
B.Comp. Thesis, School of Computing, Queen's University, April 2009

Exercise video games encourage people to be more physically active by combining entertainment and physical movement. On the other hand, conventional sedentary video games attract players with challenging mental tasks and puzzles. We address the question as to how performance on such tasks is effected by physical movement when they are placed in the context of an exercise video game.

Sports psychology research suggests a general improvement in cognitive task performance during and after physical activity. To assess if these findings can be replicated in the domain of exercise video games, we present an experiment in which we monitor the performance of game players on text comprehension, reaction time, and virtual target shooting tasks while cycling for several minutes on a recumbent bicycle. Our results suggest that the former two tasks are unaffected by exercise intensity, while the ability to perform the latter task is hampered under higher levels of exercise intensity.