Shadowing/Observation

In shadowing, the experimenter follows the participant around and records their activities.
Shadowing can occur for an unlimited time period, as long as there is a willing
participant. Closely related to shadowing, observation occurs when the experimenter
observes software engineers engaged in their work, or specific experiment-related
tasks, such as meetings or programming. The difference between shadowing and
observation is that the researcher shadows one software engineer at a time, but can
observe many at one time.
Advantages: Shadowing and observation are easy to implement, give fast results,
and require no special equipment.
Disadvantages: For shadowing, it is often difficult to see what a software engineer
is doing, especially when they are using keyboard shortcuts to issue commands and
working quickly. However, for the general picture, e.g., knowing they are now
debugging, shadowing does work well. Observers need to have a fairly good understanding
of the environment to interpret the software engineer’s behavior. This can
sometimes be offset by predefining a set of categories or looked-for behaviors. Of
course, again, this limits the type of data that will be collected.
Examples: We have implemented shadowing in our work in two ways (1997). First,
one experimenter took paper-and-pencil notes to indicate what the participant was
doing and for approximately how long. This information gave us a good general
picture of the work habits of the software engineers. We also used synchronized
shadowing where two experimenters used two laptop computers to record the software
engineer’s actions. One was responsible for ascertaining the participants’ high
level goals, while the other was responsible for recording their low-level actions.
We used pre-defined categories (Microsoft Word macros) to make recording easier.
Wu et al. (2003) also used pre-defined categories to shadow software engineers.
Perry et al. (1994) also shadowed software engineers as they went about their
work. They recorded continuous real-time non-verbal behavior in small spiral notebooks.
Additionally, at timed intervals they asked the software engineers “What are
you doing now?” At the end of each day, they converted the notebook observations to computer files. The direct observations contributed to Perry et al.’s understanding
of the software process. In particular, shadowing was good for observing informal
communication in the group setting. Similarly, Ko et al. (2007) also shadowed
software engineers. They asked the participants to think of the researchers as a new
hire to which they should explain what they were doing. From this data, they were
able to categorize the met and unmet information needs of software engineers.
As an example of observation, Teasley et al. (2002), were interested in whether
co-locating team members affects development of software. In addition to interviews
and questionnaires, they observed teams, conference calls, problem solving,
and photographed various artifacts. The researchers found that satisfaction and
productivity increased for co-located teams.
Reporting guidelines: In reporting shadowing, the precise form of shadowing and/
or observation needs to be detailed, including whether any verbal instructions were
given to the participant to think out loud. Additionally, the way the information is
recorded must be detailed as well as the length of the session, and any other special
instructions given to the participants. It is also helpful to provide context information,
such as what activities the shadowed and/or observed participants were
engaged in, and whether this was typical or not.