Think-Aloud Protocols for software engineers

In think-aloud protocol analysis (Ericcson and Simon, 1984), researchers ask
participants to think out loud while performing a task. The task can occur naturally
at work or be predetermined by the researcher. As software engineers sometimes forget
to verbalize, experimenters may occasionally remind them to continue thinking
out loud. Other than these occasional reminders, researchers do not interfere in the
problem solving process. Think-aloud sessions generally last no more than 2 hours.
Think-aloud protocol analysis is most often used for determining or validating a
cognitive model as software engineers do some programming task. For a good
review of this literature, see von Mayrhauser and Vans (1995). Additionally, if you
are considering utilizing this technique, Karahasanovic et al. (2007) provide a comprehensive
comparison of this technique to a form of work diaries, evaluating its
costs, impacts on problem solving, and benefits.
Advantages: Asking people to think aloud is relatively easy to implement.
Additionally, it is possible to implement think-aloud protocol analysis with manual
record keeping eliminating the need for transcription. This technique gives a unique
view of the problem solving process and additionally gives access to mental model.
It is an established technique.
Disadvantages: Think-aloud protocol analysis was developed for use in situations
where a researcher could map out the entire problem space. It’s not clear how this
technique translates to other domains where it is impossible to map out the problem
space a priori. However, Chi (1997) has defined a technique called Verbal Analysis
that does address this problem. In either case, even using manual record keeping, it
is difficult and time-consuming to analyze think-aloud data.
Examples: von Mayrhauser and Vans (1993) asked software developers to think
aloud as they performed a maintenance task which necessitated program comprehension.
Both software engineers involved in the experiment chose debugging
sessions. The think-aloud protocols were coded to determine if participants were
adhering to the “Integrated meta-model” of program comprehension these researchers
have defined. They found evidence for usage of this model, and were therefore
able to use the model to suggest tool requirements for software maintenance
environments.

As another example of think-aloud protocol analysis, Seaman et al. (2003) were
interested in evaluating a user interface for a prototype management system. They
asked several subjects to choose from a set of designated problems and then solve
the problem using the system. The subjects were asked to verbalize their thoughts
and motivations while working through the problems. The researchers were able to
identify positive and negative aspects of the user interface and use this information
in their evolution of the system.