Evaluating Human-Assisted Recovery Processes: Materials Recovery-Oriented Computing Research Group University of California, Berkeley

As part of the Recovery-Oriented Computing (ROC) project at UC Berkeley, we have developed an approach for evaluating the dependability consequences of human-assisted recovery processes; this approach is the first step toward human-aware recovery benchmarks.

The approach was initially developed to fill a need: we had built a human-driven recovery tool, System-wide Undo for Operators, designed to help system administrators and operators retroactively repair past errors, such as human operator errors, misconfigurations, botched software upgrades and installations, and external attacks. We wanted to evaluate the tool to understand quantitatively if and how it improved dependability, but we were unable to find any existing evaluation methodologies that fit our needs. Thus we developed a new evaluation methodology, drawing on techniques from performance benchmarks, dependability benchmarks, and usability studies.

Our technique, and the results of its use to evaluate the Undo tool, are described in the following paper:

A.B. Brown, L. Chung, W. Kakes, C. Ling, and D.A. Patterson. Experience with Evaluating Human-Assisted Recovery Processes. Proceedings of the 2004 International Conference on Dependable Systems and Networks. Florence, Italy, June 2004. [pdf]

A more detailed description and analysis can be found in the following dissertation:

A. B. Brown, "A Recovery-Oriented Approach to Dependable Services: Repairing Past Errors with System-Wide Undo." UC Berkeley Computer Science Division Technical Report UCB//CSD-04-1304, December 2003. [abstract] [pdf]

To help other researchers and practitioners who wish to use or extend our methodology, we are making available all of the materials we used to carry out our evaluation of the Undo tool. These provide more detailed insight into our study, and can serve as templates for future studies of other systems. The materials are available at the following links.

• Developing the faultload: Survey of e-mail administrators: forms and results
• Participant selection: Online screening form
• Participant training: Powerpoint slide deck for training (.ppt)
• Scenario descriptions (given to participant at start of each trial): Powerpoint slide deck (.ppt)
• Participant debriefing questionnaires (pdf): first trial (with Undo), second trial (without Undo)

All materials linked above are Copyright © 2003 The Regents of the University of California at Berkeley. They are made availabile under a Berkeley-style license; complete copyright and terms can be found here.

For questions or further information, please contact undo-dev@roc.cs.berkeley.edu.