Procedures Manual

Workspace Set-up

4 workspaces, each with a copy of modified jEdit-4.1pre6 and (unmodified) JHotDraw-6.0b1, the latest versions of the experiment plugins, the Mylar Monitor is started, and the IBM 1.4.2 JVM installed.

• plain: Eclipse R3.1.2 with only monitoring plugins.
  1. The Relevant Elements view should be open.
  2. jEdit debug profile has been run.
• jquery: Eclipse R3.1.2 with JQuery and monitoring plugins.
  1. Two working sets should be created, one for jEdit and the other with JHotDraw.
  2. The JQuery view should be opened in the same location as the Package Explorer view, with two tabs, one for JHotDraw, and the other for jEdit.
  3. The Package Explorer should be closed.
  4. The Relevant Elements view should be open.
  5. jEdit debug profile has been run.
• ariadne: Eclipse R3.1.2 with Ariadne, Ariadne JDT, and monitoring plugins.
  1. The Ariadne view should be open, and the static JDT option preselected.
  2. The Relevant Elements view should be open.
  3. jEdit debug profile has been run.
• suggestor: Eclipse R3.1.2 with Concern Mapper, Suggestions, and monitoring plugins.
  1. The Concern Mapper and Suggestions views should be open.
  2. The Relevant Elements view should be open.
  3. jEdit debug profile has been run.

Obtained printouts of:

• The Overview of the Experiment
• The Relevant Elements documentation
• The Interview Questions, 2 copies of the Consent Form, and the NASA TLX.
• The JQuery, Ariadne, Suggestor tutorials
• The task descriptions: A-T1, A-T2, B-T1, B-T2.

Prior to Participant's Arrival

1. Assign id for incoming participant, and note date and time.
2. Assign and strike configuration from configuration list.
3. Copy and configure workspace directories for respective tasks by using the c:\expt\src\configN script
4. Unzip appropriate workspaces and start first task workspace.
5. Restore window focus from follows-mouse.
6. Quit Gaim and Workrave
7. Layout 2 copies of consent form and appropriate reference card.
8. Start the first Eclipse workspace (c:\expt\task-plain\run.bat). Verify that the monitor is started and is not obfuscating references.

Administering the Experiment

Introduction (5 minutes)

1. Welcome subjects and walk through clauses of consent form: emphasizing anonimity of traces to be made public. Have them sign two copies, one for their references.
2. Have them read the Overview of Experiment page. Summarize:
   1. They will be planning two program modification tasks to jEdit. Ask if they have ever viewed the source code for jEdit.
   2. Emphasize that they are not to actually perform the changes, but rather should identify the particular classes and methods to be used, and describe any new classes required. These instructions should be appropriate for giving to a first-year coop student to rapidly turn to working code.
3. They are to record the plan using the Relevant Elements tool. Have them do the Relevant Elements tutorial. Remind them that for existing classes/methods/fields, using Control-F9; for new elements, use the new element button.
4. The investigator should situate themselves out of the participant's line-of-sight.

Task 1 (50 minutes + 4 minutes)

1. [10 minutes] Ask to read the first task description, to be performed using Eclipse:
   • Please read the task description and ask any questions.
   • We have allocated 10 minutes.
   • We cannot answer any content questions during the experiment, and only with yes/no answers.
   • Please let me know when you are ready to start the task.
2. [40 minutes] Proceed with the actual task:
   • Verify that the Mylar Monitor is running.
   • They have 40 minutes.
   • Ask if they would like updates on time.
   • They may rearrange windows as they like.
   • Note start and completion times.
   • Take field notes.
3. [4 minutes] Ask to explain what they considered.
   • Start audio recording
   • State participant id, date, and time
   • Ask to explain what they considered.
   • Stop audio recording
   • [4 minutes] Administer NASA TLX questionnaire. Note that it is one page.
     • During this time, terminate the first workspace and start the second workspace. Pre-cache information, if possible:
       • For JQuery: ensure two tabs are open on the JHotDraw and jEdit working sets. Warm up the JQuery caches for both JHotDraw and jEdit by right-clicking on respective tabs in the JQuery view's open tree browser. Ensure rules are present (triangle-menu -> reload). (3 mins approx)
       • For Ariadne: ensure Static Java operation toolbox selected.

Ask if they would like to take a break.

Task 2 (60 minutes)

1. [10 minutes] Provide copy of appropriate tool tutorial.
   • Ask to read the tutorial.
   • Show JHotDraw. Explain that JHotDraw is a simple figure editor, somewhat like Adobe Illustrator or Corel Draw, providing tools to draw, edit, and composite figures. Ask them to explain the purpose of org.jhotdraw.standard.NullTool within the JHotDraw system.
2. [10 minutes] Ask to read the first task description, to be performed using the appropriate tool:
   • Please read the task description and ask any questions.
   • We have allocated 10 minutes.
   • We cannot answer any content questions during the experiment, and only with yes/no answers.
   • Please let me know when you are ready to start the task.
3. [40 minutes] Proceed with the actual task:
   • Verify monitor is enabled.
   • Remind them that they have 40 minutes.
   • They may rearrange windows as they like.
   • Please attempt to use the tool as much as possible: we are interested in what information it cannot provide and the circumstances. If you cannot find the answer to your query, then by all means use the normal Eclipse facilities.
   • Provide updates every 10 minutes on time remaining.
   • Note start and completion times.
   • Take field notes.
4. [4 minutes] Administer NASA TLX questionnaire.

Interview (10 minutes)

1. Turn on audio recording device.
2. State participant id and date and time.
3. Conduct interview
4. Turn off audio recording device.

Conclusion

1. Create directory in results directory with participant identifier.
2. Copy the appropriate mylar-history.xml and rel-elmts-*.xml files to results directory using the c:\expt\src\store id.
3. If using Suggestor, save concern graph.
4. Clean the directories using c:\expt\src\clean