As part of the Recovery-Oriented Computing (ROC) project at UC Berkeley, we have developed an approach for evaluating the dependability consequences of human-assisted recovery processes; this approach is the first step toward human-aware recovery benchmarks.
The approach was initially developed to fill a need: we had built a human-driven recovery tool, System-wide Undo for Operators, designed to help system administrators and operators retroactively repair past errors, such as human operator errors, misconfigurations, botched software upgrades and installations, and external attacks. We wanted to evaluate the tool to understand quantitatively if and how it improved dependability, but we were unable to find any existing evaluation methodologies that fit our needs. Thus we developed a new evaluation methodology, drawing on techniques from performance benchmarks, dependability benchmarks, and usability studies.
Our technique, and the results of its use to evaluate the Undo tool, are described in the following paper:
A more detailed description and analysis can be found in the following dissertation:
To help other researchers and practitioners who wish to use or extend our methodology, we are making available all of the materials we used to carry out our evaluation of the Undo tool. These provide more detailed insight into our study, and can serve as templates for future studies of other systems. The materials are available at the following links.
All materials linked above are Copyright © 2003 The Regents of the University of California at Berkeley. They are made availabile under a Berkeley-style license; complete copyright and terms can be found here.
For questions or further information, please contact firstname.lastname@example.org.
Last modified 03 May 2005 11:22:18 -0700 . Contact: roc-group at cs.berkeley.edu.