Ducky Thesis Proposal Notes

Problem Statement

I propose to study which techniques developers use when developing code, how that varies from person to person, and how success at programming tasks correlates with choice of techniques.

I will do so via a user study that captures information about several assigned tasks:

• the questions that developers ask about the code for that task
• which IDE interactions they do to answer them
• the time that it takes to do the task
• the quality of their code, as measured by the number of unit tests which pass

From that, I will examine which techniques they use to answer which questions, and correlate the choice of technique with how fast and how well they complete the task

• the source code that they add/modify/delete

Problems:

• P1: We (software engineering researchers) do not know what different low-level techniques developers use when developing code using an IDE.
  • P1.1: We do not have a shared vocabulary for discussing different techniques that developers use when developing code with IDEs. (?)
• P2: We do not know how to recognize when people are stuck.
• P3: We do not know which techniques are the most productive.
• P4: We do not know how to teach/train developers how to be more productive.

Givens:

Jonathan Sillito showed:

• G1: Questions that developers ask in the pursuit of code can be classified.

Hypotheses:

• H0.0: We can design a study that will force people ask (generally) the same questions.
• H0.1: We can figure out what questions they are asking.
• H0.2: Developers will ask a lot of easier questions in the pursuit of answering more difficult questions. (This means that we don't have to put effort into guiding the coders to asking those easier questions.)
• H1: There are a few (and only a few) different techniques for pulling answers to a given question out of a code base via an IDE.
• H2: Which technique people use to pull information out of the source code via the IDE correlates with how fast they can at answering the question.
• H3: Which technique people use to pull information out of the source code via the IDE correlates with the quality of their solution.
• H4: Which technique people use to pull information out of the source code via the IDE correlates with how much experience they have programming.
• H5: Which technique people use to pull information out of the source code via the IDE correlates with how much experience they have with the IDE.
• H6: We can figure out when people get stuck by the patterns of their interactions with the IDE.

Literature Review

• Murphy/Kersten/Findlater
• Robillard et al
• BSD et al (unpublished)
• Jonathan Sillito's questions
• Andrew Ko ISCE05, ICSE07-tbd, CHI07-tbd, other Ko papers
• Emerson Murphy Hill tbd
• Anneliese von Mayrhauser/Andrews tbd
• Robert DeLine tbd
• Gina Venolia (at least check out again) Note to Gail: Gina has done some interesting email stuff AND a friend at Microsoft insists that I should meet her because I would like her; I can easily imagine going down to Seattle sometime and having coffee with her.
• PPIG (NB: good stuff, fertile ground)
• (programmer variability studies: Sackman/Erikson/Grant, Dickey, Curtis, DeMarco and Lister)

Proposed data-gathering methods

Three cohorts of programmers will be observed:

• Group A will be collaborators -- graduate students in CS specializing in software practices or human-computer interfaces.
• Group B will be novice programmers -- undergraduate students in CS.
• Group C will be experienced programmers -- professional code developers.

Group A will have four participants; Groups B and C will have six to ten participants each.

For each study, I will observe two people from a given cohort pair-programming -- working (and talking) together -- using the Eclipse Integrated Development Environment (IDE) to perform three to five different (small) programming assignments on a large code base.

Before the pair-programming starts, I will give the programmers a very brief survey to determine how much experience they have with programming in general, how much experience they have with Java, and how much experience they have with the Eclipse IDE.

I will videotape their interactions with each other. Their interactions with Eclipse will be logged using the Mylar Monitor, and the Mylar Monitor logs and the videotape will be synchronized.

(Note to Gail: I sort of think I'd also like to do a screen capture. Is there any downside to doing that?)

I will keep the code that they develop. While I do not forsee using the code, it is easy to keep it, minimally invasive, and potentially useful in unforseen ways.

The tasks will be designed to attempt to force the users to ask specific, relatively difficult types of questions. We expect that they will also ask a number of "easier" questions in the course of answering the more complex questions.

I will pilot the study with two pair-programming teams from cohort A. In addition to the videotaping and Mylar Monitor, I will do a semi-structured interview with the participants to gauge the effectiveness of the study.

Finally, I will run the study with at least three pair-programming teams from cohort B and three from cohort C.

Proposed analysis methods

I will watch the video specifically to look for which questions people ask, and I will put those questions into Sillito's taxonomy. I will record the questions and the times they were asked.

I will do a qualitative analysis of the logs and videotapes to see which techniques people perform after asking the questions in the service of answering those questions. I will analyze both successful techniques -- those which lead to an answer to the question.

I will also note points where the participants are "stuck", and examine the logs in an attempt to discover usage patterns that indicate when someone is stuck.

• Set ALLOWTOPICCHANGE = DuckySherwood, GailMurphy