In other research by the PI, a plug-in (MylarMonitor) was developed for the popular Eclipse IDE that can log user interactions with Eclipse. While the actual source code is not logged, MylarMonitor can log every time a developer presses a button, pulls down a menu, presses a keyboard shortcut, etc.

14. how many

1. Participants must volunteer in response to a recruitment form. @@@
2. Participants must be at least 19 years of age or older.
3. Participants must be enrolled in CPSC @@@.
4. Participants must be willing to use the Eclipse IDE for all course assignments.

1. PI

2. minimal risk

3. funding

4. where

5. title

no additional title proposed period: 2007/01/01 to 2007/05/01 @@@?

6. requirements

I need to do a consent form... DuckyEthicsConsentForm see about getting other consent forms from Gail

7. Signatures

8. Contact

9. Co-investigator

9c qualifications

9d TCPS tutorial

10 Funding source

11 peer review

12 other institutions

Background

<--  

 Set ALLOWTOPICCHANGE = DuckySherwood, GailMurphy
 
 Set ALLOWTOPICRENAME = DuckySherwood, GailMurphy
 
  Set NOSEARCHALL = on
 
 Set ALLOWWEBVIEW = DuckySherwood, GailMurphy
 
-->

We will also need cooperation from the instructor and/or teaching assistant to provide us with the assignment submission, log file, and grade for the assignment.

• Companies could use telemetry data as a factor in hiring decisions.

• Application form
• Advertisement to recruit students?
• Subject consent form
• Fee for service form?

13. Summary of research

Purpose and objective

Hypothesis

Justification

Method

Analysis

15. Subject description

Inclusion criteria: student in a computer science course which uses the Java programming language and has individually executed and graded assignments. Subjects must have at least one semester of familiarity with the the Eclipse development tool (which is already recommended) and be willing to use it again exclusively for this course.

16. Excluded

17. Approach to subjects

20. Method

22a. How much time?

• Install logger
• Task switch
• Hand in context
• Deal with crashes

23. What risks?

Long-term, if we can correlate

24. Benefits?

• improved tools

Question: do we have rights to secondary use of information from the previous usage study?

Question: who holds the id<->person mapping? Can it be destroyed after the data is submitted? Are we likely to want to do follow-up interviews?

Ducky Ethics Approval Form Draft Notes

Purpose: collect data on how students actually program in an IDE (Eclipse) and explore the data. If we find differences between the logs of high-scoring and low-scoring code development sessions, that could potentially be used to teaching and/or development tools.

Subjects: students in a computer science class which uses Eclipse/Java for individual programming assignments.

Question: which level? 200 or 300 or both? which class?

Needed from subjects: logs of their coding activity for each assignment plus the assignment handed in plus the grade that they got for the assignment.

Also needed: someone to obfuscate identities, so that we don't know who did which assignment.

Question: do we need to obfuscate what code is being modified? I don't think so, tho we probably want them to use Mylar to only give us that assignment's context.

Question: do we want to keep track of which assignments belong to which students, or have each trace/handin/grade tuple be separate?

Possible downsides for the student: slightly slower performance on Eclipse; feeling self-conscious while coding?

Compensation: ?

Possible follow-ons: look at differences between logs of professionals and students; between IDEs; between languages.

Or if the logs say that the mechanics aren't that important, that the thinking is, then maybe diverge into testing thinking ability in various ways and seeing if we can do some remedial thinking training. (?)

Potential harms:

• The monitoring plug-in could make Eclipse crash, causing them to lose work.
• If there is a recognizable difference between traces of good code development and bad code development, companies could use logging to make hire/no-hire decisions and/or performance appraisals. Companies could focus on raw coding development skills (as given by the score of the log) to the exclusion of other useful skills (like documentation, vision, quality of comments, etc).
• Companies could track their coders' work tightly enough to make the coders feel pressured and stressed. Overreliance on coding score could lead coders to "game the system" by altering their interaction patterns even to the detriment of code quality or speed.
• If instrumenting developers' tools proves successful in evaluating coders, this technique could potentially be used in other domains. While very little of a coder's personal life leaves a trace in their code, one can imagine instrumenting email as well, where there is a huge amount of personal information that routinely travels around in email.

Things to watch out for: DUCKY must make the appeal to students, NOT the instructor, and make it clear that the instructor is at most a transport mechanism.

Review: This seems like a minimal harm study, so an annual review is probably fine.

Urk. "Fewest subjects" requirement. Urk. How do we know?