Ducky Ethics Approval Form Draft Notes

DEADLINES for minimal risk studies - Nov 10, Dec 1, Dec 21

Purpose: collect data on how students actually program in an IDE (Eclipse) and explore the data. If we find differences between the logs of high-scoring and low-scoring code development sessions, that could potentially be used to teaching and/or development tools.

Subjects: students in a computer science class which uses Eclipse/Java for individual programming assignments.

Question: which level? 200 or 300 or both? which class?

Needed from subjects: logs of their coding activity for each assignment plus the assignment handed in plus the grade that they got for the assignment.

Also needed: someone to obfuscate identities, so that we don't know who did which assignment.

Question: do we need to obfuscate what code is being modified? I don't think so, tho we probably want them to use Mylar to only give us that assignment's context.

Question: do we want to keep track of which assignments belong to which students, or have each trace/handin/grade tuple be separate?

Possible downsides for the student: slightly slower performance on Eclipse; feeling self-conscious while coding?

Compensation: ?

Possible follow-ons: look at differences between logs of professionals and students; between IDEs; between languages.

Or if the logs say that the mechanics aren't that important, that the thinking is, then maybe diverge into testing thinking ability in various ways and seeing if we can do some remedial thinking training. (?)

Potential harms:

• The monitoring plug-in could make Eclipse crash, causing them to lose work.
• If there is a recognizable difference between traces of good code development and bad code development, companies could use logging to make hire/no-hire decisions and/or performance appraisals. Companies could focus on raw coding development skills (as given by the score of the log) to the exclusion of other useful skills (like documentation, vision, quality of comments, etc).
• Companies could track their coders' work tightly enough to make the coders feel pressured and stressed. Overreliance on coding score could lead coders to "game the system" by altering their interaction patterns even to the detriment of code quality or speed.
• If instrumenting developers' tools proves successful in evaluating coders, this technique could potentially be used in other domains. While very little of a coder's personal life leaves a trace in their code, one can imagine instrumenting email as well, where there is a huge amount of personal information that routinely travels around in email.

Things to watch out for: DUCKY must make the appeal to students, NOT the instructor, and make it clear that the instructor is at most a transport mechanism.

Review: This seems like a minimal harm study, so an annual review is probably fine.

Urk. "Fewest subjects" requirement. Urk. How do we know?

Question: how much do we have to tell them upfront about the purpose of the research?

Question: do we have rights to secondary use of information from the previous usage study?

Question: who holds the id<->person mapping? Can it be destroyed after the data is submitted? Are we likely to want to do follow-up interviews?

Forms needed:

• Application form
• Advertisement to recruit students?
• Subject consent form
• Fee for service form?

---

13. Summary of research

Purpose and objective

Hypothesis

Justification

Method

Analysis

15. Subject description

Students in Computer Science XXX.

Inclusion criteria: student in a computer science course which uses the Java programming language and has individually executed and graded assignments. Subjects must have at least one semester of familiarity with the the Eclipse development tool (which is already recommended) and be willing to use it again exclusively for this course.

16. Excluded

People who are not in CSXXX are excluded. Students who use a development environment other than Eclipse (approx N% last time this class was taught) are excluded.

17. Approach to subjects

Announcement in class?

20. Method

In all experiments, students will be asked to download a plug-in for Eclipse, to turn on logging when they work on their assignments for the particular class, turn off logging when they are working on a different assignment, and to hand in their log file when they hand in their assignment.

We will also need cooperation from the instructor and/or teaching assistant to provide us with the assignment submission, log file, and grade for the assignment.

22a. How much time?

• Install logger
• Task switch
• Hand in context
• Deal with crashes

23. What risks?

If the plug-in is not rock-solid stable, it could crash and cause the subject to lose work. (@@@ where do I put in that we want to make damn sure it doesn't crash)

Long-term, if we can correlate patterns in usage logs to coding performance, there are two dangers:

• Companies could use telemetry data as a factor in hiring decisions.

24. Benefits?

We hope that by participating in the study, the users will benefit indirectly from improvements in development tools and/or teaching methods that result from this study.