An examination of four user-based software evaluation methods 
Ron Henderson,  John Podd,  Mike Smith,  and Hugo Varela-Alvarez

Since the focus of the last few class periods has been evaluation, I decided to go ahead and read a paper on different  methods rather than yet another hand tracking algorithm.  The paper was written in 1995, but focuses on evaluation methods that are still used (data logging, questionnaires, interviews, 'verbal protocol analyses').

For their study, the authors got a group of 148 people and had each of them use a different evaluation method to test one of three pieces of software (spreadsheet/word processor/database).  The subjects used the software and then applied the evaluation method.

Data Logging: internal software was used to log keystrokes with time samples, to be examined after the test.

Questionnaire: used a 7 point scale with 'not applicable' and 'don't understand' options.  Questions were over topics such as

     Program  self-descriptiveness

     User  control  of  the  program 

    Ease  of  learning  the  program  

    Completeness  of  the  program 

    Correspondence  with  user  expectations

    Flexibility  in  task  handling  

    Fault  tolerance 

    Formatting 

Open ended questions followed these asking about specific problems and calling for comments/suggestions.

Interview: Semi-structured format of scripted questions and following up on unique interviewee comments.

Verbal Protocol: Video taped users while they were evaluating the software.  Users were later asked to 'think aloud' as they watched the tapes play back.

Conclusions:

Data logging is nice because it's pretty much as objective as you can get, however, it's tedious to analyze.

Questionnaires can be give vague results if the wording is not incredibly specific for each question, and it's difficult to make questionnaires that everybody will understand completely.

Interviews are good for getting relevant information quickly, but are subject to the problem of memory decay.

The verbal protocol method tends to be good at finding problem areas because it calls to memory when the user was having trouble with a particular exercise.  However, it's very time-consuming.

The authors note that using a combination of these methods will most likely give the best results, as they add unique contributions, but that using multiple methods is probably affected by diminishing returns, so just blindly adding more methods is not the best approach.

Source: http://ac.els-cdn.com/0953543896877010/1-s2.0-0953543896877010-main.pdf?_tid=dc537806ba2ff563889e04ee57c57acb&acdnat=1332869815_55b023c54f66c39a1a8e626956eabd53