CrossCulturalReview < Main

---+C-TOC - Cross-Cultural Review Panel
Notes by Matthew Brehmer, M.Sc Researcher, UBC Dept. of Computer Sc.

%TOC%

---++ First Iteration: May 27, 2010

---+++ Introduction

   * Setup of C-TOC prototype on Douglas College computer lab: no security/admin issues - macros working successfully
   * refer to literacy education for seniors / immigrants - largely computer-based
   * refer to UBC learning exchange w/ DT east side (apparently 40yo = senior)
   * Claudia's introductory presentation
   * cultural advisory coordinators: Kymberley Bontinen, Patricia Juvik, Sai Roshni (Priya) Raju; CHCP (centre for health and community partnerships) project manager: Marina Niks
   * cultural advisory panel members (5, one missing)
      * Chinese, Japanese, South Asian, Southeast Asian, Latin American communities represented
      * Vivian Lam, Sayuri Sugawara, Kamaljit Kaur, Lam Dang, Karla Maranhao
      * recruited from community groups that interact w/ seniors in their respective communities
      * representative of health-care / extended care related positions, nursing, counselling
      * all members of the panel are immigrants

---+++ Interactive Prototype Session Notes

   * users: 5 panel members, M. Niks, R. Hsiung, S. Raju, K. Bontinen, P. Juvik (10 total)
   * difficulty realising what is a mockup and what is interactive (i.e. drop-down menus in introductory slides) - distinction may not be clear; 
      * suggestion: use a warning label in the future?
   * red prompts for referring to questionnaire not obvious enough - centre on screen rather than top-left? 
      * suggestion: include pictures representative of each test on the questionnaire as reminders / in case they have skipped ahead and forgotten the questionnaire
   * drag-and-drop habit hard to shake - everyone had difficulty with click-and-release moving of objects - some skipped the intro slide "move the blue circle into the square"; difficulty arose in sentence comprehension, which depended on this interaction; by this point many had developed an understanding of this interaction
      * will older users be similarly biased / used to traditional drag-and-drop? it requires constant motor pressure - did the panel understand this rationale for using click-and-release?
      * suggestion: try out traditional drag-and-drop in next prototype version and ask what is preferred
   * pattern construction instructions not well understood by some users; dragging to other areas of the screen aside from the target zone; for instance, covering the source shape;
      * one user puts all the component shapes back into the source area after completing the target shape; not sure why?
   * clicking on trails test not required but done by most users
      * suggestion: state that it is not necessary, provide animated instruction sequence
   * click+drag on a shape misaligns the cursor, decoupling it from the shape causing confusion and the possibility for the shape to be dropped off-screen, or being unable to click-to-release shape unless the cursor is clicked on top of the shape;
      * not much to be done - a fault of the PPT macro
   * square puzzles not fully interactive - some lines are still missing the drag/drop macro (#7 in particular)
   * "stack on top" instruction in sentence comprehension has potential for multiple interpretations: some users layer shapes, some place shapes above one another on screen
   * some go/no-go slides not advancing / hyperlink is broken and leads to an erroneous place in the slide deck
   * sentence production instructions are still missing the additional instruction to "use as many words as possible"
   * computers low on memory after opening 16 slide decks (will be resolved in non-ppt version)
   * most participants finished session in 1h20min; final participant in 1h30min ("too long!")

---+++ Focus Group Notes

   * Could handwriting be used as a diagnostic tool?
   * general consensus: this test will work well for well-educated, high-income, well-integrated people
   * problem w/ mouse drag-and-drop were vocalised
      * suggestion: avoid this by using touch-screens
   * request for audio instructions as optional feature for all test instructions; other instruction formats could include a flow diagram / animated step-by-step instructions
   * more practice sessions requested
   * language throughout the tests needs to be overhauled, cultural-specific terms discarded; possible to translate into other languages?
      * Similarities test especially sensitive to language norms
      * cultural-specific names in item recognition test misleading (i.e. "stationery item")
      * suggestion: allow family members to translate, but not help with task
   * provide an "I don't understand" option, a flag to de-validate a single test if necessary
   * community centre use case scenario: group members help one another take the test rather than family/caregivers (peer support)
      * identified shame in cognitive illness, acknowledging it in the household, esp. in South Asian community
   * suggestion: provide feedback on each test: descriptive time elapsed, # items correct, where you scored comparatively to pop. norms, past times taken
      * how to deliver potentially bad news over the net? instead direct to community resources regardless of test performance

   * next panel meeting (late October 15/28/29?)
      * suggestion: provide a shorter evaluation version of the test for cultural advisory panel (i.e. less trials per test), provide web-accessible version of next prototype version in October, allow them to take + fill out survey from home - likely not in PPT, but in interactive programmed version (Flex?)
      * longer focus group next time, allow discussion on interactivity of each test
      * more focus group members to be recruited

---++ Second Iteration: October, 2010

---+++ Introduction

   * Setup of C-TOC prototype on Douglas College computer lab: no security/admin issues - macros working successfully
   * refer to literacy education for seniors / immigrants - largely computer-based
   * Claudia's introductory presentation
   * cultural advisory coordinators: Kymberley Bontinen, Patricia Juvik, Sai Roshni (Priya) Raju; CHCP (centre for health and community partnerships) project manager: Marina Niks
   * cultural advisory panel members (5, one missing)
      * Chinese, Japanese, South Asian, Southeast Asian, Latin American communities represented
      * Vivian Lam, Sayuri Sugawara, Kamaljit Kaur, Norma Sanchez, Gita Rafiee
      * recruited from community groups that interact w/ seniors in their respective communities
      * representative of health-care / extended care related positions, nursing, counselling

---+++ Interactive Prototype Session Notes

   * users: 6 panel members, M. Niks, S. Raju, K. Bontinen, P. Juvik (10 total)
   * observers: MB, HL, CJ, CT
   * session started ~40 min late, finished ~45 min behind schedule (longest participant session 1h15min, shortest around 45min)

   * intro/help menu/throughout CTOC
      * tooltips on buttons and links may be confusing
      * help menu has no exit button (must click on 'help' again to exit
      * by moving "back" from slide 2 to intro screen, you are unable to move forwards again;
      * too much text in help dialogs
      * pull-down menus not yet interactive
      * red questionnaire response cues are being missed - not obvious enough
      * "Click" here to proceed to next screen on page 1 - ambiguous as to where to click  
      * yellow dot to click on unresponsive for some users
      * "are you still there?" prompt premature, timeout too soon 

   * PWP - after clicking on stimulus once, there is a long pause before feedback is given - potential source of spurious clicking or confusion
      * inform user of pause? mouse-over the name - only require 1 click? 
      * on "correct" screen - show PWP before next trial?

   * SDM - stopwatch image confusing - small clicking target - long pauses here

   * SC - inconsistency in feedback given after practice problems: "correct" vs. "good"
      * "overlapping" vs. "on top"
      * obvious drag/drop issue

   * SIM - no pictures for "fame" and "money" stimuli

   * WR - no dedicated questionnaire page - tied to PWP

   * MOS - need to click twice on misplaced object
      * would "click on object that doesn't belong" work?
      * contrast problem - potential need for hiring a graphic designer to design scenes and stimuli
      * bigger objects required

   * TRA - automatic generation of lines on mouse-over confusing and likely invalid - people will click anyways;
       * numbers and letters culturally dependent

   * !SqP - one trial has impossible instruction
      * obviously difficult (for healthy adults)

   * PC - wording instructions too small
      * reconstruction: "... like these" : instead of "."

   * SP - dra/drop macro missing for some stimuli

   * GNG - instructions too lengthy, not being re-read while they are subtly different

---+++ Focus Group Notes

   * demographics: acceptability of CTOC in 80+ - computer literacy issues
   * children will bring CTOC home to parents - dependent on culture and living situations - others will prefer company of peers in public spaces - knowledgeable peers or nurses
      * high trust and confidentiality among peers in some communities, others among family
   * knowledge of English still an issue; more graphical instructions needed to address language issue; 
      * provide a limited version w/ no language used, full version w/ language?
   * high level of education, computer literacy, cultural knowledge required - can cultural content be more general?
   * test could agitate depression or severe anxiety; can people break at will? a clear way to pause the test needed; test takes a lot of energy, very fatiguing
   * consider a practice version of the test to be taken before undertaking the full version?
   * adjusting test difficulty dynamically dependent on early performance - increasing difficulty may conflict w/ fatigue - confounding variables
      * "sorry try again" discontinuation rules after consecutive failures
   * multiple clicks and pauses a hassle, spurious clicking - must strike a balance b/w always needing to click and not clicking
   * moving objects very difficult
   * variable scoring based on location of test-taking and current anxiety score?

   * practice problems / instructions:
      * buttons top small throughout
      * use audio for instructions and feedback but not for test content - may be a distraction - optional audio
      * graphical sequence requested - demonstrating low level mouse clicks - accessible via help menu
      * graphical instructions well received, increased confidence and reduced stress - clear pictures
      * could instructions affect test validity? biases?
      * provide multiple options for different kinds of learners
      * itemise instructions to encourage re-reading of instructions with subtle differences (i.e. Go/No-Go)

   * interruptions / distractions
      * physical noise - preventative, doctor's office scenario? use of sound-cancelling headphones? could still need to shut out loud conversation
      * resumption cues: could provide an automated summary, recap illustrations, one-step back
      * strategy: resume scoring on next trial - several cutoffs for "are you still there?" several restart points dependent on length of interruption 
      * bathroom breaks should be considered

---++ Third Iteration: February, 2011

---+++ Introduction

   * Setup of C-TOC prototype on Douglas College computer lab: no security/admin issues - macros working successfully
   * UBC attendees: MB, RH, HL, CL, WW (CJ ill)
   * Robins's introductory presentation
   * cultural advisory coordinators: Kymberley Bontinen, Patricia Juvik, Sai Roshni (Priya) Raju, Marjory Ditmars; CHCP (centre for health and community partnerships) project manager: Marina Niks
   * cultural advisory panel members (4, one missing)
      * South Asian, Southeast Asian, Persian, Latin American communities represented
      * Kamaljit Kaur, Norma Sanchez, Gita Rafiee, Fariyal Dhirani
      * recruited from community groups that interact w/ seniors in their respective communities
      * representative of health-care / extended care related positions, nursing, counselling

---+++ Interactive Prototype Session Notes

   * users: 4 panel members, CHCP coordinators (6 total)
   * observers: MB, HL, CL, RH, WW
   * longest participant session 1h15min, shortest around 1hr
   * delayed start (surveys, headphones needed to be distributed)

   * General / Navigation 
      * button affordances required throughout (mouseOver, mouseDown, disabled feedback, audible click on mouseDown); 
         * ppt hyperlink mouseOver dialogs are shown currently (confusing, meaningless text is shown)
      * consistent button UI: colour, text colour, gradients, border, font, font size, bevel, etc.
         * many nav. buttons have poor-contrast black text on blue background. 
         * help buttons are different from nav. buttons
         * I suggest a permanent control bar with small set of icons for help and navigation, visible throughout the test; pause button, progress bar visible between tests;
      * click to pick up, drag, and click to drop instructions still insufficient; default drag-drop behaviour overrides; will be eliminated in programmed version;
      * computer-literate-biased terminology used throughout: "Back", "Icon", potentially more?

   * Introduction
      * drop-down menus still not implemented (explained by small dialog)

   * Help

   * Picture-Word Pairs

   * Symbol-Digit Matching

   * Similarities

   * Word Recognition 

   * Pattern Construction

   * Sentence Comprehension

   * Trails

   * Arithmetic

   * Pattern Reconstruction

   * Misplaced Object Search

   * Sentence Production

   * Square Puzzles

   * Go No-Go

   * Misplaced Object Recall

   * Conclusion

---+++ Focus Group Notes

   * acceptability

   * context of use

   * language

   * general usability

   * audio

   * practice problems / instructions:

   * interruptions / distractions

---
-- Main.MatthewBrehmer - 01 Jun 2010
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	CulturalAdvisoryMeetingCycle2.pdf	r1	manage	105.2 K	2010-10-21 - 21:33	MatthewBrehmer	Cultural Advisory panel 2 feedback
pdf	CulturalAdvisoryPanelMinutesOctober15.pdf	r1	manage	201.2 K	2010-10-21 - 21:33	MatthewBrehmer	Cultural Advisory panel 2 minutes
This topic: Main > TWikiUsers > MatthewBrehmer > C-TOC > CrossCulturalReview
Topic revision: r5 - 2011-02-05 - MatthewBrehmer