Table of Contents

Title Page

Copyright Page

Table of Figures

PREFACE

Acknowledgements

THE AUTHOR

CHAPTER 1 - AN INTRODUCTION TO PUBLIC AND COMMUNITY HEALTH EVALUATION

LEARNING OBJECTIVES

THE LINKS AMONG COMMUNITY ASSESSMENT, PROGRAM IMPLEMENTATION, AND EVALUATION

OVERVIEW OF EVALUATION

PREASSESSMENT EVALUATIONS

COMMUNITY-BASED PARTICIPATORY RESEARCH

THE PARTICIPATORY MODEL FOR EVALUATION

CULTURAL CONSIDERATIONS IN EVALUATION

SUMMARY

DISCUSSION QUESTIONS AND ACTIVITIES

KEY TERMS

CHAPTER 2 - THE COMMUNITY ASSESSMENT

LEARNING OBJECTIVES

THEORETICAL CONSIDERATIONS

THE ECOLOGICAL MODEL

DATA COLLECTION

PROCESS

DATA SOURCES

REVIEWING THE SCIENTIFIC LITERATURE

THE REPORT

STAKEHOLDERS’ PARTICIPATION IN COMMUNITY ASSESSMENTS

SUMMARY

DISCUSSION QUESTIONS AND ACTIVITIES

KEY TERMS

CHAPTER 3 - DEVELOPING INITIATIVES

LEARNING OBJECTIVES

THE ORGANIZATION’S MISSION

PLANNING THE INITIATIVE

GOALS AND OBJECTIVES

THE INITIATIVE ACTIVITIES

USING EXISTING EVIDENCE-BASED PROGRAMS

THE PROGRAM’S THEORY OF CHANGE

THE LOGIC MODEL DEPICTING THE THEORY OF CHANGE

CRITERIA FOR SUCCESSFUL INITIATIVES

SUMMARY

DISCUSSION QUESTIONS AND ACTIVITIES

KEY TERMS

CHAPTER 4 - PLANNING FOR EVALUATION

LEARNING OBJECTIVES

THE TIMING OF THE EVALUATION

THE PURPOSE OF EVALUATION

ESTABLISHING THE CONTRACT FOR EVALUATION

THE EVALUATION TEAM

CREATING AND MAINTAINING EFFECTIVE PARTNERSHIPS

EVALUATION STANDARDS

MANAGING THE EVALUATION PROCESS

Deciding on Contract Terms, Supervision, and Finances

FACTORS THAT INFLUENCE THE EVALUATION PROCESS

INVOLVING STAKEHOLDERS

SUMMARY

DISCUSSION QUESTIONS AND ACTIVITIES

KEY TERMS

CHAPTER 5 - DESIGNING THE EVALUATION

LEARNING OBJECTIVES

THE CONTEXT

JUSTIFICATIONS FOR THE INITIATIVE

THE INITIATIVE’S GOALS, OBJECTIVES, AND ACTIVITIES

THE INITIATIVE’S THEORY OF CHANGE AND LOGIC MODEL

SUMMARY

DISCUSSION QUESTIONS AND ACTIVITIES

KEY TERMS

CHAPTER 6 - DESIGNING THE EVALUATION

LEARNING OBJECTIVES

BASES FOR SELECTING THE EVALUATION QUESTIONS

APPROACHES TO SELECTING THE EVALUATION QUESTIONS

TYPES OF EVALUATIONS

PROCESS EVALUATION

OUTCOME EVALUATION

Measuring Concepts and Constructs

SUMMARY

DISCUSSION QUESTIONS AND ACTIVITIES

KEY TERMS

CHAPTER 7 - COLLECTING THE DATA

LEARNING OBJECTIVES

CHOOSING A DATA-COLLECTION APPROACH

USING SURVEYS

DESIGNING SURVEY INSTRUMENTS

PILOT TESTING

TRIANGULATION

INSTITUTIONAL REVIEW BOARDS

STAKEHOLDER INVOLVEMENT

SUMMARY

DISCUSSION QUESTIONS AND ACTIVITIES

KEY TERMS

CHAPTER 8 - ANALYZING AND INTERPRETING THE DATA

LEARNING OBJECTIVES

ANALYZING AND REPORTING QUANTITATIVE DATA

REACHING CONCLUSIONS

STAKEHOLDER INVOLVEMENT

STEPS IN QUANTITATIVE-DATA ANALYSIS AND INTERPRETATION

SUMMARY

DISCUSSION QUESTIONS AND ACTIVITIES

KEY TERMS

CHAPTER 9 - COLLECTING THE DATA

LEARNING OBJECTIVES

QUALITATIVE DATA

ENSURING VALIDITY AND RELIABILITY

INTERVIEW-FORMAT APPROACHES

DOCUMENT AND RECORD REVIEW

OBSERVATIONAL APPROACHES

CASE REVIEWS

DIGITAL APPROACHES

GEOGRAPHIC INFORMATION SYSTEMS

TRAINING DATA COLLECTORS

PILOT TESTING

MANAGING AND STORING QUALITATIVE DATA

STAKEHOLDER INVOLVEMENT

SUMMARY

DISCUSSION QUESTIONS AND ACTIVITIES

KEY TERMS

CHAPTER 10 - ANALYZING AND INTERPRETING THE DATA

LEARNING OBJECTIVES

ANALYZING QUALITATIVE DATA

INTERPRETING THE DATA AND REACHING CONCLUSIONS

THE ROLE OF STAKEHOLDERS

SUMMARY

DISCUSSION QUESTIONS AND ACTIVITIES

KEY TERMS

CHAPTER 11 - REPORTING EVALUATION FINDINGS

LEARNING OBJECTIVES

THE CONTENT OF THE REPORT

THE TIMING OF THE REPORT

THE AUDIENCE FOR THE REPORT

THE FORMAT OF THE REPORT

SUMMARY

DISCUSSION QUESTIONS AND ACTIVITIES

KEY TERMS

CHAPTER 12 - CASE STUDY

THE COMMUNITY ASSESSMENT

THE INTERVENTION

DESIGN THE EVALUATION

COLLECT THE DATA

ANALYZE AND INTERPRET THE DATA

REPORT THE RESULTS

GLOSSARY

REFERENCES

INDEX

Table of Figures

FIGURE 1.1. Evaluation in Context

FIGURE 1.2. Framework for Program Evaluation in Public Health

FIGURE 1.3. The Participatory Model for Evaluation

FIGURE 2.1. The Community Assessment as a Component of the Participatory Model for Evaluation

FIGURE 2.2. The Community Assessment, the Initiative, and the Evaluation

FIGURE 2.3. The Ecological-Model Framework

FIGURE 2.4. Framework for Summarizing a Literature Review

FIGURE 3.1. The Hierarchical Relationship of Goals and Objectives

FIGURE 3.2. Verbs for Writing Objectives

FIGURE 3.3. Components of Evidence-Based Programs

FIGURE 3.4. Classification System for Evidence-Based HIV/ AIDS Interventions

FIGURE 3.5. Basic Framework for a Logic Model

FIGURE 5.1. The Participatory Framework for Evaluation: Design the Evaluation

FIGURE 5.2. Organizational Chart

FIGURE 5.3. Logic Model for a Program to Train Older Adults as Companions

FIGURE 6.1. Logic-Model Framework for Identifying Evaluation Questions

FIGURE 6.2. Relationship Between Outcome and Activity Objectives and Outputs

FIGURE 6.3. Two-by-Two Table

FIGURE 6.4. Performance of the Indicator Following an Intervention

FIGURE 6.5. Mean Scores Before and After an Intervention

FIGURE 7.1. The Participatory Framework for Evaluation: Collect the Data

FIGURE 8.1. The Participatory Framework for Evaluation: Analyze and Interpret the Data

FIGURE 8.2. Bar-Chart Format for Presenting Quantitative Data

FIGURE 8.3. Pie-Chart Format for Presenting Quantitative Data

FIGURE 10.1. Screen Shots of Qualitative-Data Coding from NVivo QSR

FIGURE 11.1. The Participatory Framework for Evaluation: Report the Findings

FIGURE 12.1. Theory of Change

FIGURE 12.2. Two-by-Two Table

Copyright © 2010 by John Wiley & Sons, Inc. All rights reserved.

Published by Jossey-Bass
A Wiley Imprint
989 Market Street, San Francisco, CA 94103-1741—www.josseybass.com

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax 978-646-8600, or on the Web at www.copyright.com. Requests to the publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax 201-748-6008, or online at www.wiley.com/go/permissions.

Readers should be aware that Internet Web sites offered as citations and/or sources for further information may have changed or disappeared between the time this was written and when it is read.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

Jossey-Bass books and products are available through most bookstores. To contact Jossey-Bass directly call our Customer Care Department within the U.S. at 800-956-7739, outside the U.S. at 317-572-3986, or fax 317-572-4002.

Jossey-Bass also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

Library of Congress Cataloging-in-Publication Data

Harris, Muriel J., 1955-

p. ; cm.

Includes bibliographical references and index.

eISBN : 978-0-470-58619-8

1. Public health—Evaluation. 2. Community health services—Evaluation. I. Title. [DNLM: 1. Community Health Services—standards. 2. Program Evaluation—methods. 3. Community-Based Participatory Research. 4. Data Collection—methods. 5. Evaluation Studies as Topic. W 84.41 H315e 2010]

RA440.4.H37 2010

362.1072—dc22

2009044622

PB Printing

PREFACE

In my early career as a public health practitioner I headed a community-based organization that focused on HIV/AIDS prevention for women. I administered lots of surveys to determine the differences in participants’ knowledge before and after educational programs. During that period I also assessed a different kind of intervention. In the months leading up to World AIDS Day, we often heard that women felt unable to talk about condoms with their partners and therefore were unable to protect themselves from becoming infected with the virus that causes AIDS. My organization wanted to change that situation and declared World AIDS Day a “No Sex Day,” using the print and electronic media to provide information to the public. We encouraged the discussion of HIV/AIDS and abstinence from sex on that day but the use of condoms if couples decided to have sex.

We wanted to know whether our activities leading up to World AIDS Day had any effect on those who heard the programs on radio or television or read articles about our activities in the newspapers. We wanted to know whether women were able to discuss HIV/AIDS with their partners, whether they were able to abstain from having sex, and whether they had used condoms. We wanted to know whether the program effect was different for single women than for married women. We developed a short survey with five simple questions. We did not call what we did an evaluation. It did not have a name. It was just called a questionnaire!

A couple of years later, soon after I completed a situation analysis of homeless children, I was asked to design a drop-in center for this population and to ensure that the project had an evaluation component. Designing the drop-in center was easy. But evaluation was an unfamiliar term to me at the time, and I had lots of questions. What is evaluation? How do you evaluate a drop-in center? How do you know whether you are making a difference for homeless children? Where do you start? I had prepared questionnaires before and after my activities in the past. This evaluation, though, would be different and would test the limits of my experience.

Not long after that and now many years ago, I embarked on a path to learning what that term meant and to developing skills for program evaluation with the support of many wonderful mentors. I now teach program evaluation and take great pleasure in sharing my thoughts and ideas with my students as many of them struggle to learn this challenging subject. As you embark on your journey to learn about evaluating programs, I trust that this book will guide you. You may not know what the term evaluation means, and, like me all those years ago and many of my students now, you are probably still a little wary. No matter where you are in your understanding, my hope is that whether you are a practitioner, a student, or both, you will find this book helpful on your journey and on your path to understanding.

Just as I did many years ago, you probably evaluate what you do all the time without giving it a name. Evaluation is often an unconscious activity that is carried out before choosing among one or many options, both informally and formally. Informal evaluations range from selecting a restaurant for dinner to selecting a course of dishes off the menu. All the decisions you make along the way have implications for the success or failure of the outing. At the end of the evening, you go over the steps you took and decide whether the trip was worth it. If it wasn’t, you may decide never to go to that restaurant again. So it is with program evaluation. We assess the resources and activities that went into a program, and then we determine whether the program was worth it to those who participated and to those who funded it.

Evaluation activities occur in a range of work-related settings including community-based organizations, coalitions and partnerships, government-funded entities, the pharmaceutical industry, and the media. Program evaluations assess how an event or activity was conducted, how well it was conducted, and whether it achieved its goal. Evaluation determines the merit of a program or policy, and it forms the basis for decision making.

Evaluation is the cornerstone of program improvement and must be carefully planned and executed to be effective. It helps make the task of assessing the appropriateness of a public health intervention or the success of a program or policy explicit by using appropriate research methods. In evaluation a plan is developed to assess the achievement of program objectives. The plan states the standards against which the intervention will be assessed, the scope of the evaluation, and appropriate tools and approaches for data collection and analysis.

As a further illustration of the evaluation process, let’s consider the story of a twenty-four-year-old female who lives in a community not far from where she works. After hearing a lot of talk about the importance of exercising, Antoinette, who works for an evaluation research organization, reached the conclusion that it was a good thing to do, and she decided to check out the neighborhood to see where she could exercise. She knew that she needed to conduct an assessment. She went to the local phone book and found two exercise facilities in her neighborhood.

The following day Antoinette drove by both the gyms and liked what she saw on the outside, but she needed additional information about each facility. She knew that she needed to have a good idea of what she wanted in an exercise facility to be able to make a good judgment. As a result, she sat down and made a list that contained three major categories: the structure and context, content, and expected benefits. Armed with her list of must haves, which included a well-equipped, safe facility with knowledgeable staff and satisfied customers, she set off to see both facilities. As she approached each facility she looked around at her surroundings and noted the type of building; as she entered she thought about the amount of lighting and the resources that had been put into the facility. She talked to the staff at the desk, the manager, and a trainer. She looked at the equipment and the training programs offered, and finally she talked to a few customers about their experiences and asked whether using the facility had helped them. After a long day Antoinette went home to consider her options. The next morning Antoinette reviewed all the information she had gathered and compared it with her list of must-haves. She compared each facility against her criteria. After analyzing all the data she had collected, she decided that the exercise center that more closely matched her criteria was the best and was the one she would join.

The following week Antoinette set off for the Regent Avenue Gym. She filled out the lengthy membership form, paid her dues, and started her daily exercise routine. Antoinette continually evaluated the program and her surroundings. She monitored the service to determine whether it continued to meet her expectations and all her criteria. After a few weeks in the program Antoinette felt a lot better than she had. She was able to run up stairs, walk longer distances, and overall she just felt better. Her doctor told her that exercising regularly would make her healthier and lower her risk of disease. Antoinette continued to exercise regularly for many years.

Antoinette used a number of approaches to evaluate the Regent Avenue Gym. No one approach or methodology for evaluation fits all situations; so it is important to match the method with the program or policy initiative being evaluated. The stakeholders, who have an interest in the program and the outcome of the evaluation, may influence the type of questions asked, the criteria that are selected, and the evaluation methodologies that are chosen.

There are many opportunities to conduct an evaluation during the life of an intervention, and the approaches to conducting the evaluation in each case will differ. The methods and tools for an evaluation that is conducted during the first few months of a program are different from those used when the program or participation in the program ends and the effectiveness of the program or policy is being assessed. In addition, during the life of the program, evaluation tools and approaches can be used to record program and policy participation and progress.

Evaluation is essentially about making judgments of worth; an evaluation assesses a program’s success in achieving the objectives that were established in the beginning. In addition it assesses how well the program was run, who was exposed to the program or policy, and how they benefited from the intervention.

This book presents a model for evaluation and describes the approaches and methods for evaluating community health program and policy interventions. It is aimed at public health and community health students as well as practitioners who are new to program and policy evaluation. This book makes no assumptions of prior knowledge about evaluation. The approach to evaluation that is presented allows for the development of simple or complex evaluation plans while focusing on practical approaches. It encourages a critical thinking and reflective approach with the full involvement of multiple stakeholders throughout the evaluation process. This book provides learners with a systematic, step -by-step approach to program evaluation.

The book is organized into twelve chapters. It discusses the community assessment and the development of the public health initiative as the precursors to the four-step participatory model for evaluation with stakeholders at the center of each component. It frames program evaluation in the context of Community-Based Participatory Research. A case study concludes the book. Additional forms, worksheets, and resources are available for downloading on the publisher’s website for the book.

ACKNOWLEDGMENTS

I would like to extend my appreciation to my family, who have supported me in innumerable ways. To D.J. and Benny, I hope this will inspire you to pursue your dreams. Thank you to my mentors, who taught me my first lessons in evaluating programs. Thank you to the organizations that allowed me to develop my evaluation skills through helping others learn evaluation. In the course of doing what we love best, we learn so much. I am grateful that I had the chance to learn from my students, whose patience in learning evaluation through their group projects is a constant inspiration. Thank you to my friends and colleagues, who have continued to encourage me, and to those who reviewed the manuscript and provided important and valuable input and feedback. To my publisher, this dream may never have come true without you. I thank you most sincerely. To each of you who reads this book, I hope you will find much pleasure in learning evaluation.

THE AUTHOR

Muriel J. Harris, PhD, MPH, is an assistant professor at the University of Louisville, School of Public Health and Information Sciences. She also works with the Seattle-based Center for Community Health and Evaluation, part of the Group Health Center for Health Studies, on Kentucky-based evaluation projects.

She developed her interest and her initial practice in public health program evaluation while working in Liberia and Sierra Leone in West Africa. She received her formal training at the Arnold School of Public Health, University of South Carolina. While at the Institute for Families in Society, University of South Carolina, she developed her skills in Empowerment Evaluation and conducted evaluations of a variety of public health programs.

Dr. Harris is a strong supporter of participatory evaluation. In her teaching she supports students as they get real-world experiences from conducting evaluations with community partners. Dr. Harris received the 2008 Delta Omega award for innovative curriculum in public health.

CHAPTER 1

AN INTRODUCTION TO PUBLIC AND COMMUNITY HEALTH EVALUATION

LEARNING OBJECTIVES

• Describe the links among community assessment, program implementation, and program evaluation.

• Describe preassessment evaluations.

• Identify the uses and approaches of evaluation.

• List the principles of Community-Based Participatory Research.

• Explain the ethical and cultural issues in evaluation.

• Describe the value and role of stakeholders in evaluation.

Public health may be assessed by the impact it has on improving the quality of life of people and communities through the elimination or the reduction in the incidence, prevalence, and rates of disease and disability. It should improve conditions and access to resources for healthy living for all people. Public health programs and policies may be instituted at the local, state, national, or international level.

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!

Lesen Sie weiter in der vollständigen Ausgabe!