Instrumenting Systems for software engineers

This technique requires “instrumentation” to be built into the software tools used
by the software engineer. This instrumentation is used to record information automatically
about the usage of the tools. Instrumentation can be used to monitor how
frequently a tool or feature is used, patterns of access to files and directories, and
even the timing underlying different activities. This technique is also called system
monitoring.
In some cases, instrumentation merely records the commands issued by users.
More advanced forms of instrumentation record both the input and output in great
detail so that the researcher can effectively play back the session. Others have proposed
building a new set of tools with embedded instruments to further constrain
the work environment (Buckley and Cahill, 1997). Related to this, Johnson and his
group have developed Hackystat, an open-source server-based system for monitoring
actions. Developers install sensors on their machines that then relay information
to a centralized server (see www.csdl.ics.hawaii.edu/Research/hackystat for more
information).

Advantages: System monitoring requires no time commitment from software
engineers. Since, people tend to be very poor judges of factors such as relative frequency
and duration of the various activities they perform, this technique can be
used to provide such information accurately.
Disadvantages: It is difficult to analyze data from instrumented systems meaningfully;
that is, it is difficult to determine software engineers’ thoughts and goals from
a series of tool invocations. This problem is particularly relevant when the working
environment is not well understood or constrained. For example, software engineers
often customize their environments by adding scripts and macros (e.g., in
emacs). One way of dealing with this disadvantage is to play back the events to a
software engineer and ask them to comment. Although in many jurisdictions,
employers have the right to monitor employees, there are ethical concerns if
researchers become involved in monitoring software engineers without their
knowledge.

Examples: Budgen and Thomson (2003) used a logging element when assessing
how useful a particular CASE tool was. The logger element recorded data whenever
an event occurred. Events were predetermined before. Textual data was not
recorded. The researchers found that recording events only was a shortcoming of
their design. It would have been more appropriate to collect information about the
context of the particular event.
As another example, Walenstein (2003) used VNC (Virtual Network Computing)
to collect verbatim screen protocols (continuous screen captures) of software developers
engaged in software development activities. Walenstein also collected verbal
protocols and used a theory-based approach to analyse the data.
More recently, Storey et al. (2007) logged developers’ use of their TagSEA tool.
The logs were stored on the client machine. The software engineers downloaded
them to a server at specified intervals. The logs enabled Storey et al. (2007) to
understand how the tool was being used, and nicely complemented other data
sources such as interviews and a focus group. Similar to this study, Zou and
Godfrey (2007) used a logger to determine which artifacts software maintainers
were just viewing, and which were actually changed.
Reporting guidelines: The precise nature of the logging needs to be reported, including
any special instrumentation installed on the software engineer’s machines. This should
include a description of what exactly is logged, with what frequency. Any special considerations
with respect to data processing and analysis should also be detailed.