C-TOC Usability Evaluation - Iteration 1

Interview 1

Details

• Date/time: Monday, May 17, 2010, 10am - 12:30pm
• Location: Brain Science Research Centre, Koerner pavilion, Exam Rm #9
• Interviewers: Claudia (clinical), Matt (HCI)

• Interviewee:
  • age/gender: 76 / M
  • profession: physics professor (retired) (still gives guest lectures)
  • diagnosis: NCI (has memory complaints, esp. for remembering names)
  • other: cataracts, poor eyesight
  • computer literacy: atypically high
    • has used a Mac for research since early 80's (views as research tool only)
    • currently owns 28" iMac at home, another 24" iMac at the office, and a laptop;
    • has only ever used a one-button Mac mouse (left-handed)
    • uses Adobe Illustrator to create figures for academic publications / presentations
    • uses Google as research tool
    • uses MS Word, email, safari web browser, spreadsheets
    • no games, frivolous web browsing (viewed as waste of time)
    • self-taught computer user (skill transfer from type-writer)
    • relies on friends / grandsons for resolving computer issues; doesn't rely on help documentation

Notes

• smaller font size on instruction screens
• intro. checklist too much
• show task completion / progress (i.e. "slide x of y")
• break up sentence clauses on to new lines
• PPT slideshow semitransparent navigation buttons are misleading and conflict w/ C-TOC navigation buttons
• wording disliked: "battery", "practice"
• why is the test / task necessary? what does it assess? otherwise it is viewed as frivolous nonsense
• tests are mostly fun/easy
• unaware of implied instructions to remember word/picture pairs
• sentence comprehension: difficulty remembering instructions
• trails: requires 2 reads of instructions
• misplaced objects recall: poor performance

• afterwards, eye strain, fatigues (2.5 hours)
• better to have examiner present; would not want to take online
• test is too long (known), interacting w/ computers seen as artificial, wouldn't be as fun alone, seeming like a chore

test fixes:

• symbol-digit matching: clicking on number fields not working, only clicking on numbers
• similarities: instructions at top of test
• sentence comprehension: instructions should be on screen
• sentence production: mechanically moving words not functional
• go-stop - slides not advancing; useless

not yet interactive / implemented

• sentence comprehension
• sentence production
• pattern construction
• pattern recall
• square puzzles
• go-stop

Interview 2

Details

• Date/time: Thursday, June 3rd, 2010, 10am - 12:30pm
• Location: Brain Science Research Centre, Koerner pavilion, Exam Rm #9
• Interviewers: Claudia (clinical), Matt (HCI)

• Interviewee:
  • age/gender: ?? / M
  • profession: former VP-finance of large corporation (retired)
  • diagnosis: MCI - mild AD - short term memory impairment evident (MMSE = 23)
  • other: wears glasses
  • computer literacy: some experience (modest)
    • first used a computer ~20 years ago at work; took classes - used for interpretation of results, reading documents, prepared financial statements, business correspondence, recalls using Lotus
    • currently owns a desktop PC running windows, claims to be 'archaic', which he bought himself
    • right-handed, familiar with a mouse
    • ~10 hours a week of computer use/week (generous), mostly games (FreeCell and Scrabble); not the highlight of his week, but some enjoyment is had
    • occasionally responds to email (ranging from 1/month to 1/week)
    • doesn't use computers elsewhere
    • familiar with MS Word
    • doesn't use web browsers mentioned (IE / Firefox - confused about question)
    • doesn't rely on help documentation

Notes

• intro: first slide: missed bubble in bottom right to proceed to next slide; clicked anywhere expecting to proceed
• current date on temporal orientation test: May 18, spring begins on May 28?
• symbol-digit matching: unsure initially how to proceed
• memory tasks (word recognition, misplaced object recall, pattern recall) significantly impaired, sentence comprehension suffered due to memory impairments
• pattern construction: square inside triangle corners result was odd and very spaced-out compared to target
• arithmetic appropriate for his background (finance), likely challenging for most others
• mouse drag-and-drop habit hard to shake at times - would likely need to adopt traditional drag-and-drop formalism
• square puzzles: perhaps add a counter, keeping track of the # of lines already moved
• go-stop test purpose not clear: "am I supposed to do something?"
• reflection: enjoyed challenges, unsure about accessing on the web (would need help from companion), can't recall help menu or what he would add to one, would need help setting up the test

test fixes:

• technical observations and/or suggested quick C-TOC fixes:
  • throughout: next/prev buttons on all slides for consistency (except slides in which it is states "click anywhere to continue")
  • symbol-digit matching test: instructions to click on response bar need to be clearer; there may be a confusion with the source bar
  • pattern construction test: instructions on where to pick up shapes and where to drop shapes need to be clearer; potential confusion resulting from attempting to pick up pieces of the target shape;
  • trails test: it is not clear that clicking is not required in this test
  • sentence comprehension test: too many test items;
    • mouse glitches resulting from drag operation (conflicts with the point-and-click move macro) - we identified this problem at Douglas College;
    • 'smallest to largest' cue used in one trial not visible enough
  • misplaced object search test: include a 'give up' or 'I don't know' button to proceed to next trial
  • sentence production test: slide advancement on first 'click anywhere to begin' slide glitches (likely a memory + PPT issue);
    • mouse glitches resulting from drag operation - specifically dropping problems (conflicts with the point-and-click move macro) - we identified this problem at Douglas College;
    • still missing a 'use as many words as possible' instruction - were we planning on adding this after our meeting with Barbara?
  • square puzzles test: practice trial is not given instructions, but test trials are given instructions;
    • presently difficult to grab lines (another hassle of the point-and-click move macro) - could we widen the lines slightly to facilitate grabbing? would this have any impact on the results? (i.e. from 1pt to 3pt)
  • go-stop test: some slide advancement problems (likely a memory + PPT issue again)
  • misplaced-object recall test: instructions not clear w.r.t. the object being misplaced vs. contextually appropriate?;
    • some slide advancement problems (likely a memory + PPT issue again)

Interview 3

Details

• Date/time: Friday, June 4th, 2010, 10am - 12:30pm
• Location: Brain Science Research Centre, Koerner pavilion, Exam Rm #9
• Interviewers: Claudia (clinical), Matt (HCI)

• Interviewee:
  • age/gender: 71 / F
  • profession: retired medical transcriber
  • diagnosis: mild AD + on anti-psychotic medication to alleviate delirium (MMSE = 20)
    • problems with attention, balance, hearing
  • other: wears glasses
  • education: Capilano college, Political science
  • computer literacy: low
    • an experienced typist, developed CT syndrome
    • did use a computer many years ago, floppy disk, transcriptions for doctors, inkjet printer
    • learned computer use through a combination of means
    • has a computer at home in disuse ("in limbo" - email full, automatic updates a hassle)
    • not familiar with a mouse, no longer uses a computer at all

Notes:

• withdrew after 2 tests (picture-word pairs test and word-recognition test)
• would need companion present to take test / take in a controlled lab setting

test fixes:

• technical observations and/or suggested quick C-TOC fixes:
  • make nav. buttons larger
  • add "see prev. screen" to help menu where available/possible
  • picture-word pairs test: instructions to click/advance needed on all slides where currently missing
    • some ambiguous / non-prototypical object images
  • word recognition test: "click on numbered choice" instruction too small, to be moved to top of screen

Interview 4

Details

• Date/time: Thursday, June 17, 2010, 10am - 12:30pm
• Location: Brain Science Research Centre, Koerner pavilion, Exam Rm #8
• Interviewers: Claudia (clinical), Matt (HCI)

• Interviewee:
  • age/gender: 68 / F
  • profession: retired nurse / hospital shift supervisor
  • diagnosis: FTD (early stage)
    • problems with distraction/attention, balance, tingling in extremities (fingers/toes - likely related to Diabetes)
  • other: wears glasses - early cataracts - otherwise fine
  • education: B.Sc nursing
  • computer literacy: somewhat - not much use of computers currently (no longer needs to use most applications) - seen as very useful, quite enjoys using a computer
    • experienced due to work-related computer use; retrieving patient records, scheduling, biomedical services, organising staffing - once offered a manager position in which she would teach computer use skills to other staff members at Kewlowna Gen. Hospital
    • now: reads emails infrequently, sometimes will send an email, views photos sent to her and her husband; husband helps with computer use
    • not familiar with majority of computer programs - no web browsing, document writing, at least not recently
    • once borrowed her daughter's Wii system, became bored really fast, not amused
    • husband has laptop at home, 2 button mouse, Windows 7
  • positive reaction to test, hopeful for computerised testing; "good idea"; happy to be tests on a computer, open to most things nowadays being on a computer

Notes:

• excellent performance on most tests (more difficulty with spatial/language tests - sentence production, square puzzles, pattern construction (not recall)
• help menu fine, nothing comes to mind
• most tests seen as easier than pen+paper version, more intuitive, no differences expected at home, makes more sense to be done on a computer
• overall difficulty with instruction screens; not clear what to expect without practice version or demonstration (audio/visual tutorial?) - felt like "diving in blind" to many of the tests, despite performing well after completing trial / practice items - instructions clear then
• drag/drop vs. point/click - no preference "whatever works"
• willing to take test again - est. 1/2 hour to an hour
• tests were fun
• would be likely distracted at home, mind would wander (FTD symptom), CG would need to be in room to keep on task; motivation using a timer/clock image and/or progress bar

test fixes/suggestions:

• Temporal Orientation: "I don't know / guess" button? - unable to proceed without guessing
• Symbol Digit Matching: number ~ digit terminology - wording consistency
  • "click to continue" vs "click for next trial" unambiguous wording to facilitate the option of taking a break
• Similarities : "e.g." / example given in instruction slide
• Pattern Construction: 2 colours misleading - more than 2 colours? different for each trial shape?
• Sentence Comprehension: automatic correct screen rather than "continue/done" button
  • "start w/ smallest" instruction is a memory aid? remove? alter?
• Arithmetic: "Arithmetics" is a typo? no"s"?
• Sentence Production: make target drop zone for sentences more obvious
• Square Puzzles: need "start over" option to work
• Misplaced Object Recall: instruction unclear - attempted to drag misplaced item to corresponding image, rather than selecting the scene (step 1) and selecting the location (step 2)

• the entire battery: clear "click to proceed" / "click to go back" buttons on all slides where this is missing / unclear, especially in Picture-Word-Pairs - currently many clicks are unprompted

Interview 5

Details

• Date/time: Friday, June 25, 2010, 11am - 1:00pm
• Location: Brain Science Research Centre, Koerner pavilion, Exam Rm #9
• Interviewers: Claudia (clinical), Matt (HCI)

• Interviewee:
  • age/gender: 71/F
  • profession: physician / paediatric psychiatrist (retired) - background in marketing research during medical school
  • diagnosis: slight CIND, induced by fall + concussion
    • fatigue-induced memory, language, and coordination issues, no visuospatial concerns, no impaired judgment,
  • other: post-concussion eyesight loss (some double-vision)
  • education: medical school
  • computer literacy: claims to not be very computer-literate
    • learned while in Barbados, 1995; self-taught
    • performs "basic stuff, like email", since return from Barbados
    • has a Mac laptop at home with external keyboard (usually takes with her wherever she goes; does not use other computers), iPhone (but only uses it for placing calls)
    • 1/hr a day for computer use, mostly email and web browsing
    • uses MS Word, web browser, email, Excel (mostly used by secretary), iPhoto (incl. photo editing)
    • secretary handles most difficult tasks / applications; hates drop-down menus in MS Windows
    • doesn't enjoy using computers, viewed as a necessity, a tool
  • reaction to test: positive, 30 min fine (fatigue being largest threat)

Notes:

• "click to pick up; move; click to drop" is "pretty stupid" (tried to do regular drag-and-drop despite instruction); should this be clearer somehow? we should likely abandon point-and-click;
  • potentially a good test to see if user is reading carefully or just skimming
• help menu fine
• "highest grade achieved" ambiguous
• tests should potentially be easier at home due to comfort in one's own environment
• TO: need a different arrangement of dates in grid view to be shown
• SDM: clicking directly on numbers required, numbered boxes not yet interactive (PPT limitation?)
• SIM: automated navigation / slide advancement would be intuitive, "like the Mac"
• PC: "click to pick up; move; click to drop" still frustrating, but getting used to it; PC difficult due to diagnosis
  • "Done" bttn unnecessary, expecting automated advancement b/w trials;
  • easier on a computer, but sometimes forgot instructions, tried as fast as possible
• Trails: people will try to click anyways; ignore? force a click?
• Arithmetics: "better than counting backwards by sevens"
• PR: "?" distracting, have nothing on screen instead
• MOS: "don't understand, have to go back to beginning" "I'm not supposed to click "give up""?
  • use automated advance or prompt to click?
  • tendency to drag/drop
  • after clicking, big object appears: "not sure what to do next"
• SP: couldn't edit the sentence once started, sentences short, assembled quickly; would want to move / select multiple words, like on a Mac
  • initially occluded other word in the grid
  • not sure what it is measuring
• SqP: macro bug (difficulty dropping, keeping it in place)
• GS: confusing title - mouse moved away during target word to get next word to appear quicker; instructions unclear? "could be made harder"
• MOR: tendency to drag misplace object into scene; potentially into drop zone?
• overall:
  • willing to take test again
  • include emotional perception test
  • would like to know that results will not be shown before test being
  • would shut out everyone from house, all distractions to set up testing environment;

test fixes/suggestions:

• provide more practice using "click to pick up; move; click to drop" or use conventional drag-and-drop
• inform that results will not be shown at end before test being
• SP: add ability to place words between existing words, move multiple words at once
• throughout: automated slide advances or prompts to advance; visible navigation buttons at all times;
• Trails: force a click to move to next symbol
• GS: changed Go-Stop name

Interview 6

Details

• Date/time: Friday, June 25, 2010, 2:30pm - 4:30pm
• Location: Brain Science Research Centre, Koerner pavilion, Exam Rm #9
• Interviewers: Claudia (clinical), Matt (HCI)

• Interviewee:
  • age/gender: 69/M, married
  • profession: president of college (retired), admin in post-secondary institutions
  • diagnosis: CIND
    • executive dysfunction, fell off ladder in 2002; stability in second test, modest visuospatial ability loss, gain in some other abilities on 2nd screening
    • bipolar disorder (lithium as a AD protective)
  • other: tremor in RH (slight)
  • education: Master's in Ed. Admin
  • computer literacy: basic computer use
    • first used at work in 1986, used for nothing eventually, took some classes as VP of college
    • 2 desktops at home running Win XP, Office 2007, uses everything in that package
    • would use computer when travelling, comfortable with public computers
    • uses computer every day, office software, internet browsing; Word, Explorer, Outlook, Excel, views photos, shoutcast radio
    • computer use "strictly business"
    • consultant for website redesign; "most fun in past 2 years"
    • relies on wife for help with computers, trial and error; experienced problems with Office 2007, but hasn't given up;
    • computers are essential; would feel lonely without a familiarity with computers
  • reaction to test: computerised tests a blessing; willing to take test depending on structure, 5hr tests made him tired, no stress in 2nd screening; 30 min appropriate;

Notes:

• intro: "next" cloud on first screen, thought to click there
  • also has conventional drag-drop habit
• PWP: would need additional prompts to click forward
  • stuck at beginning, wrt instructions
  • would be more comfortable at home, performing better on test and recall
• SDM: had to trace mouse from prompt to response
  • "this is the way to go", more comfortable
• SIM: instructions should be above options
  • thought that no feedback was a mistake in design, less comfortable and reassuring, more difficult
• WR: clicking on text is first instinct, not in #'d box
• PC: difficulty w/ puzzles - bug in practice test - instruction box cannot be used as drop zone
  • "target" should read "desired", "reproduce blue pattern w/ green shape"
  • a clock would induce anxiety
• SC: "correct" message reassuring
  • note enough room for lining shapes up from smallest to largest (vertical rather than horizontal);
  • "I felt doubt", forgets instructions
• Trails: automatic line appearance unexpected, tried to drag lines
  • notion of speed took him off guard
  • done before on paper
• MOS: didn't realise object would reveal itself
  • provide lures and +/- feedback for context-appropriate items?
  • pleasant to see actual feedback
• SP: "finding this very difficult", not a word person
• GS: would be exciting to have a stopwatch on screen;
  • bug after "Sorry, Try Again" - broken links
  • appropriately challenging, but in a fun way
• MOR: difficult without feedback
• overall: provide opportunity to go back a page;
  • images were pleasant, could have had more
  • "perfectly OK", on edge of fatigue
  • provide a "thank you slide", a reward in finishing

test fixes/suggestions:

• intro: more practice w/ "click to pick up, move, click to move"
• PC: fix instruction box drop zone bug in practice trial
• Trails: force clicking rather than mouse-over?
• MOS: context-appropriate lures?
• GS: fix broken link bug after "Sorry, Try Again"
• throughout: ability to go back a page, where appropriate or permitted
• overall: a "thank-you slide"

Interview 7

Details

• Date/time: Monday, July 6, 2010, 10am - 12:00pm
• Location: Brain Science Research Centre, Koerner pavilion, Exam Rm #8
• Interviewers: Claudia (clinical), Matt (HCI)

• Interviewee:
  • age/gender: 81/M
  • profession: farming / sales (retired)
  • diagnosis: ?
  • other: memory complaints, hearing aids, no longer driving, difficulty reading
  • education: no post-secondary, Irish grammar school until 15yo
  • computer literacy: none (never before used a computer), wife has computer at home
    • used his thumb to click left mouse button
  • reaction to test: positive

Notes:

• problem with visual field - difficulty reading instructions
• spoke aloud instructions
• needed glasses to interact with test
• difficulty clicking nav buttons
• difficulty clicking check boxes in introduction
• more dragging practice necessary
• obviously needed more prompts to click to continue
• SDM: started clicking on all the responses in turn (didn't see instructions), attempted to click on symbol itself
• Sim: instruction bar too small / didn't see this
• Sim: "Sorry, Try Again" screen is ambiguous - how to proceed?
• WR: difficulty clicking directly on the response number
• PC: attempted to click on blue shape rather than green
• Trails: "I' perceived as "1"
• Discontinued after Trails
• Expected computer to help him when stuck
• Needs more practice with mouse

test fixes/suggestions:

• See usability report for suggestions