Data Collection Techniques survey

We use
the taxonomy to organize the presentation of the techniques, beginning with direct
techniques, moving on to indirect techniques, and concluding with independent
techniques. Each of the techniques is described in the same way. First the technique
is described. Then its advantages and disadvantages are identified. Next, one or
more examples of its use in software engineering research are given. Finally, some
guidance is given regarding special considerations when reporting the technique

Direct Techniques

The first five techniques listed in Table 1 are what we call inquisitive techniques
(brainstorming, focus groups, interviews, questionnaires, conceptual modeling),
while the remaining ones are primarily observational. Each type is appropriate for
gathering different types of information from software engineers.


Table 1
Questions asked by software engineering researchers (column 2) that can be answered
by field study techniques
Used by researchers Also used
when their goal is Volume by software
Technique to understand: of data engineers for
Direct techniques
Brainstorming Ideas and general Small Requirements
and focus background about gathering, project
groups the process and product, planning
general opinions
(also useful to enhance
participant rapport)
Interviews and General information Small Requirements
questionnaires (including opinions) to large and evaluation
about process, product,
personal knowledge etc.
Conceptual Mental models of Small Requirements
modeling product or process
Work diaries Time spent or frequency of certain Medium Time sheets
tasks (rough approximation,
over days or weeks)
Think-aloud Mental models, goals, Medium UI evaluation
sessions rationale and patterns to large
of activities
Shadowing and Time spent or frequency of tasks Small Advanced
observation (intermittent over relatively approaches to
short periods), patterns of use case or task
activities, some goals and analysis
rationale
Participant Deep understanding, goals and Medium
observation rationale for actions, time to large
(joining the spent or frequency over
team) a long period
Indirect techniques
Instrumenting Software usage over a long Large Software
systems period, for many participants usage analysis
Fly on the wall Time spent intermittently in one Medium
location, patterns of activities
(particularly collaboration)
Independent techniques
Analysis of work Long-term patterns relating to Large Metrics
databases software evolution, faults etc. gathering
Analysis of Details of tool usage Large
tool use logs
Documentation Design and documentation Medium Reverse
analysis practices, general engineering
understanding
Static and dynamic Design and programming Large Program
analysis practices, general comprehension,
understanding metrics, testing

Inquisitive techniques allow the experimenter to obtain a general understanding
of the software engineering process. Such techniques are probably the only way to
gauge how enjoyable or motivating certain tools are to use or certain activities to
perform. However, they are often subjective, and additionally do not allow for
accurate time measurements.
Observational techniques provide a real-time portrayal of the studied phenomena.
However, it is more difficult to analyze the data, both because it is dense and
because it requires considerable knowledge to interpret correctly. Observational
techniques can be used at randomly chosen times or when a software engineer is
engaged in a specific type of activity (such as whenever she is using a debugger).
Observational techniques always run the risk of changing the process simply by
observing it; the Hawthorne (Draper, 2004; Robbins, 1994) effect was first identified
when a group of researchers found that output was not related to environmental
conditions as expected, but rather to whether or not workers were being observed.
Careful consideration of this effect is therefore warranted in implementing the
research and explaining its purpose and protocol to the research participants.