Health Informatics (

(1)

Health Informatics

( formerly Computers in Health Care)

Kathryn J. Hannah Marion J. Ball

Series Editors

(2)

James G. Anderson Carolyn E. Aydin

Editors

Evaluating the

Organizational Impact of Healthcare

Information Systems

Second Edition

With 48 Illustrations

(3)

Library of Congress Control Number: 2005923548

ISBN 10: 0-387-24558-8 Printed on acid-free paper.

ISBN 13: 978-0387-24558-4

All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, Inc., 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden.

The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identiﬁed as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.

While the advice and information in this book are believed to be true and accurate at the date of going to press, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.

Printed in the United States of America. (BS/MVY) 9 8 7 6 5 4 3 2 1 SPIN 10972649

springeronline.com

James G. Anderson, PhD Carolyn E. Aydin, PhD Professor of Medical Sociology Research Scientist

Professor of Health Communication Nursing Research and Development Fellow, American College of Cedars-Sinai Medical Center and Burns

Medical Informatics and Allen Research Institute Co-Director of the Rural Center Los Angeles, CA 90048

for AIDS/STD Prevention USA

Department of Sociology and Anthropology Purdue University

West Lafayette, IN 47907 USA

Series Editors:

Kathryn J. Hannah, PhD, RN Marion J. Ball, EdD

Adjunct Professor, Department Vice President, Clinical Informatics of Community Health Sciences Strategies

Faculty of Medicine Healthlink, Inc.

The University of Calgary Baltimore, MD 21210

Calgary, Alberta T2N 4N1 USA

Canada and

Adjunct Professor

The Johns Hopkins University School of Nursing

Baltimore, MD 21205 USA

(4)

Series Preface

v

This series is directed to healthcare professionals who are leading the transformation of health care by using information and knowledge to advance the quality of patient care. Launched in 1988 as Computers in Health Care, the series offers a broad range of titles: some are addressed to speciﬁc pro- fessions such as nursing, medicine, and health administration; others to special areas of practice such as trauma and radiology. Still other books in the series focus on interdisciplinary issues, such as the computer-based patient record, electronic health records, and networked healthcare systems.

Renamed Health Informatics in 1998 to reflect the rapid evolution in the discipline now known as health informatics, the series continues to add titles that contribute to the evolution of the field. In the series, eminent experts, serving as editors or authors, offer their accounts of innovation in health informatics. Increasingly, these accounts go beyond hardware and software to address the role of information in influencing the transformation of healthcare delivery systems around the world. The series also increasingly focuses on “peopleware” and the organizational, behavioral, and societal changes that accompany the diffusion of information technology in health service environments.

These changes will shape health services in the new millennium. By making full and creative use of the technology to tame data and to trans- form information, health informatics will foster the development of the knowledge age in health care. As coeditors, we pledge to support our pro- fessional colleagues and the series readers as they share the advances in the emerging and exciting ﬁeld of health informatics.

Kathryn J. Hannah, PhD, RN Marion J. Ball, EdD

(5)

Preface

Information systems pervade complex organizations. In healthcare organizations such as hospitals, the U.S. Congressional Ofﬁce of Technology Assessment has estimated that computerized systems, when fully implemented, account for 4% to 8% of an institution’s total operating budget.

As healthcare costs continue to spiral upward, healthcare institutions are under increasing pressure from purchasers and payers of services to create a cost effective system by controlling operating costs while maintaining quality of care and service. The Institute of Medicine also estimates that as many as 98,000 deaths occur each year because of medical errors.

Information systems are being marketed to healthcare organizations to provide management information, control costs, facilitate total quality management and continuous quality improvement programs, and improve patient safety. Cost control and improvements in safety and quality are the two major premises on which decisions to purchase information systems are based.

There is mounting evidence, however, that the implementation of many information systems has resulted in unforeseen costs, unfulfilled promises, and disillusionment. There is also the growing realization that information systems affect the structure and functioning of organizations, the quality of work life of employees within them, and ultimately the cost and quality of the goods and services they provide. Professionals who develop, implement, and evaluate clinical computer systems, however, frequently address only the technical aspects of these systems, while the success of implementation and utilization depends upon integration of the computer system into a complex organizational setting. Without an evaluation strategy that goes beyond the technical aspects of the system, an institution has no means of knowing how well it is actually functioning within the organization and no firm basis for developing specific interventions to enhance system success.

Including these issues in systems evaluations will increase the likelihood of implementing a system that is cost effective for the organization as a whole.

The purpose of this book is to provide computer system developers, administrators, healthcare policy analysts, chief information ofﬁcers, inves-

vii

(6)

tigators, and others with a guide for evaluating the impacts of computerized information systems on (1) the structure and functioning of healthcare organizations, (2) the quality of work life of individual healthcare professionals and others working within the organization, and (3) the cost- effective delivery of health care. Evaluating information system impacts requires not only an understanding of computer technology, but also an understanding of the social and behavioral processes that affect and are affected by the introduction of this technology into organizational settings.

Investigators in the social sciences have developed theoretical foundations and analytical approaches to help understand the impact and use of information systems, but few guidelines exist to help developers, administrators, and evaluators design evaluation strategies and select appropriate methods to study system outcomes.

This book is designed as a practical guide for determining appropriate questions to ask and the most effective methods available to answer those questions. The book begins with the premise that any evaluation must be preceded by a clear statement of study objectives. Next, investigators should recognize their own perspective and assumptions concerning how information systems affect and are affected by the organizational setting in which they are implemented. Only at this point are investigators ready to review and select appropriate methodologies to answer their research questions.

The selection of appropriate methodologies is critical to the successful outcome of any investigation. Given the complex interrelationships between computer systems and their organizational environments, there is no one best method for evaluation. Rather, the selection of methods will be determined by the evaluation objectives. This book advocates a pluralistic approach, providing the reader with detailed information on a number of methods that can be used to evaluate healthcare information systems. More than one evaluation strategy may be brought to bear on the same problem domain, with each method providing a different, complementary view of the issues under study. The book is designed to assist an investigator in selecting among different methods to build the speciﬁc approach that will be most fruitful for investigating a given situation or problem. The chapters also provide a practical overview of established research guidelines for sampling, data collection procedures and instruments, and analytic techniques.

The material presented in this book draws on more than two decades of empirical studies in healthcare computing conducted by the contributors and others. Individual chapters review speciﬁc methods for organizational evaluation such as direct observations, use of archival data, interviewing strategies, survey research, cognitive approaches, work sampling, simulation, and social network analysis. Part I begins with an overview of theoretical perspectives and evaluation questions, followed by eight chapters covering different methods for evaluating the impacts of information viii Preface

(7)

systems using examples speciﬁc to healthcare organizations. Each of the eight chapters provides the reader with a detailed overview of a speciﬁc method, followed by annotated references at the end of the chapter for further reading. The example studies in Part II illustrate different evaluation methods and provide the reader with an understanding of the nature and scope of evaluation research and its importance in studying the impact of information systems, including providing information for practical decision-making and interventions.

The book also draws from a variety of social science disciplines to integrate the study of information systems with social science theory and methods. We argue that investigators in the social sciences have developed theoretical frameworks and analytical approaches that can help understand how the introduction of computer systems in healthcare settings affect the quality of the work environment, tasks and skills of health professionals, social interactions among professionals in the organization, and the effective delivery of medical care. We hope to make the developers and users of medical information systems more aware of (1) the extent to which the success of these systems depends upon complex social processes, and (2) the contributions the social sciences can make in helping to understand these processes.

The study of information systems, however, also requires social scientists themselves to develop new theories, data collection techniques, and analytic methods. This book should provide investigators and students with a start- ing point for new theoretical and policy oriented research into the impact of information systems on healthcare organizations. We also hope to initi- ate a dialogue between adherents of different research approaches, helping to clarify the range of methods and their appropriateness, strengths and weaknesses, and the understanding that can be acquired by combining different methods in a single research endeavor.

Finally, there is growing awareness at colleges and universities of the importance of studying and evaluating the use and impact of information systems as evidenced by the growth of curricula and faculty positions in the information sciences; medical, dental, and nursing informatics; and healthcare administration. Moreover, some schools are developing joint teaching and research programs that draw from diverse disciplines such as medicine, computer science, information systems, library sciences, organizational behavior, operations, management, and social sciences. This book is meant to provide a useful guide to the research and evaluation of systems for this wide variety of disciplines as well as to system developers, administrators, and practitioners.

We wish to thank Marilyn M. Anderson for her assistance in coordinat- ing and assembling the various contributions to this book.

James G. Anderson, PhD Carolyn E. Aydin, PhD Preface ix

(8)

Acknowledgments

The editors gratefully acknowledge that the following material has been reprinted with kind permission.

Chapter 7 reprinted from:

Sittig, DF. Work-Sampling: a statistical approach to evaluation of the effect of computers on work patterns in healthcare. Methods Inf Med 1993;32:167–74.

Chapters 8 and 9 reprinted from:

Computers in Biology and Medicine 32(3):151–164, 179–193. Anderson JG.

Aydin, Carolyn E., Anderson, James G., Rosen, Peter N., Felitti, Vincent J., Weng, Hui-Ching. “Computers in the Consulting Room: a case study of clini- cian and patient perspectives.” Health Care Management Science 1 (1998) 61–74.

Balzer Science Publishers BV.

Massaro, Thomas A. Introducing Physician Order entry at a Major Academic Medical Center 1 and 2. Academic Medicine. Vol. 68, No. 1, January 1993, pp. 20–30.

Journal of the American Medical Informatics Association, V9(5):479–490, Anderson JG et al: “Evaluating the capability of information technology to prevent adverse drug events: A computer simulation approach.” © 2002 American Medical Informatics Association.

Aydin, C.E., & Forsythe, D.E. (1997). Implementing Computers in Ambulatory Care: Implications of Physician Practice Patterns for System Design. Proceed- ings of the 1997 AMIA Fall Symposium, Nashville, TN, pp. 677–681. (Journal of the American Medical Informatics Association, Symposium Supplement. © 1997 American Medical Informatics Association.)

Korst, Lisa M, Eusebio-Angeja, Alea C., Chamorro, Terry, Aydin, Carolyn E., Gregory, Kimberly D. “Nursing Documentation Time during Implementation of an Electronic Medical Record.” Jrnl of Nsg Admin 2003;33(1):24–30.

Andrews, Robert D., Gardner, Reed M., Metcalf, Sandy M., Simmons, Deon.

“Computer Charting: An Evaluation of a Respiratory Care Computer System.”

Respiratory Care, August 1985, Vol. 30, No. 8, pp. 695–708.

x

(9)

Series Preface v Preface vii Contributors xiii

PART I EVALUATING HEALTHCARE INFORMATION SYSTEMS: A MULTIMETHOD APPROACH

Chapter 1 Overview: Theoretical Perspectives and Methodologies for the Evaluation of Healthcare Information Systems 5

James G. Anderson and Carolyn E. Aydin

Chapter 2 Qualitative Research Methods for Evaluating Computer Information Systems 30

Bonnie Kaplan and Joseph A. Maxwell

Chapter 3 Multiple Perspectives: Evaluating Healthcare Information Systems in Collaborative Environments 56

Madhu Reddy and Erin Bradner

Chapter 4 Survey Methods for Assessing Social Impacts of Computers in Healthcare Organizations 75

Carolyn E. Aydin

Chapter 5 Using the Internet for Surveys and Research 129 Gunther Eysenbach

Chapter 6 Cognitive Approaches to the Evaluation of Healthcare Information Systems 144

Andre W. Kushniruk and Vimla L. Patel

Chapter 7 Work-Sampling: A Statistical Approach to Evaluation of the Effect of Computers on Work Patterns in Health Care 174

Dean F. Sittig

Contributors

xiii

(12)

xiv Contributors

Gunther Eysenbach, MD

Senior Scientist, Centre for Global eHealth Innovation, Division of Medical Decision Making and Health Care Research, Toronto General Research Institute of the UHN, Toronto, ON M5S 1A1, Canada; Associate Professor, Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON M5S 1A1, Canada

Vincent J. Felitti, MD

Co-Principal Investigator, Adverse Childhood Experiences Study; Clinical Professor of Medicine, Department of Medicine, University of California, San Diego, San Diego, CA 92093, USA

Diana E. Forsythe, PhD (deceased)

Formerly Associate Adjunct Professor, Department of Medical Anthropology, University of California, San Francisco, San Francisco, CA 94143, USA

Reed M. Gardner, PhD

Professor and Chairman, Department of Medical Informatics, University of Utah, Salt Lake City, UT 84112, USA

Kimberly D. Gregory, MD, MPH

Director of Maternal Fetal Medicine and Women’s Health Services Research, Cedars-Sinai Medical Center, Los Angeles, CA 90048; Associate Professor-in-Residence, Department of Obstetrics and Gynecology, David Geffen School of Medicine, University of California, Los Angeles, Los Angeles, CA 90095, USA

Thaddeus J. Hunt, JD

Law Ofﬁce of Thaddeus Hunt, Chicago, IL 60603, USA Stephen J. Jay, MD

Professor and Chair, Department of Public Health, School of Medicine, Indiana University, Indianapolis, IN 46202, USA

Bonnie Kaplan, PhD

Lecturer, Yale Center for Medical Informatics, Yale University School of Medicine, New Haven, CT 06520; President, Kaplan Associates, Hamden, CT 06517, USA

Lisa M. Korst, MD, PhD

Associate Professor, Department of Pediatrics, Division of Research on Children, Youth, and Families, Childrens Hospital Los Angeles, Los Angeles, CA 90027, USA

(13)

Contributors xv Andre W. Kushniruk, PhD

Associate Professor and Director, School of Health Information Science, University of Victoria, BC V8W 2Y2, Canada

Thomas A. Massaro, MD, PhD

Director of Performance Improvement, University of Virginia Medical Center, Charlottesville, VA 22908; Associate Dean for Graduate Medical Education and Professor, Department of Pediatrics, School of Medicine, University of Virginia, Charlottesville, VA 22908, USA

Joseph A. Maxwell, PhD

Associate Professor, Graduate School of Education, George Mason University, Fairfax, VA 22030, USA

Sandy M. Metcalf, RRT, BS

Quality Consultant for Surgical Services, Urban Central Region, Intermountain Health Care, Salt Lake City, UT 84111, USA

Vimla L. Patel, PhD

Professor, Department of Biomedical Informatics, Columbia University, New York, NY 10032; Professor, Department of Psychiatry, New York State Psychiatric Institute, New York, NY 10032; Adjunct Professor of Psychology and Education, Columbia Teachers College, New York, NY 10027, USA Madhu Reddy, PhD

Assistant Professor, Department of Information Science and Technology, School of Management and Information Systems, University of Missouri, Rolla, MO 65409, USA

Peter N. Rosen, MD

Director, Laboratory for Visual Performance Assessment, University of California, San Diego, San Diego, CA 92093, USA

Deon Simmons, RRT

Assistant Director, Department of Respiratory Care, LDS Hospital, Salt Lake City, UT 84143, USA

Dean F. Sittig, PhD

Director of Applied Research in Medical Informatics, Northwest Permanente, Portland, OR 97232, USA

Hui-Ching Weng, PhD

Assistant Professor, Department of Health Management, I-Shou University, Taiwan 840

Health Informatics (