Assignment: Health Communication Interventions

Assignment: Health Communication Interventions
Assignment: Health Communication Interventions
Permalink:
Assignment: Evaluating Outcomes of Health Communication Interventions
Chapter 14
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
*
Overview/Objectives
Why, What and How We Measure
Evaluation Models and Tips
Case Studies
Linking Outcomes to Strategic Communication Programs
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
Why, What and How We Measure
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
The Language of Evaluation
Evaluation
Program Assessment
Metrics
Return on Investment (ROI)
Outcomes
Impact
Social Change Indicators and Social Impact
Behavioral Impact
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Social Change and Behavioral Indicators
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013, figure 14.1. © Copyright 2013. All rights reserved.
Why We Measure
Focus staff/partners/intended audiences on shared goals
Clarify program purposes, goals, and objectives
Identify and compare effective practices
Improve service delivery
Adjust program in progress by refining strategies/messages
Assess cost effectiveness of program
Assess program results/ROI
In public health communication, ROI = health or social behavior that leads to improved public health outcomes
Determine program reproducibility and sustainability/opportunities for scaling-up phase
Communicate results to key stakeholders/ audiences
Implement lessons learned from new models/strategies in future interventions
Compete for economic and human resources
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Drawbacks of Evaluation
Cost
Time
Chance of measuring wrong variables and indicators
Questionable accuracy for programs with limited scope, reach and duration
Potential bias in evaluation method or tool
Hard to do if not planned ahead
Results that may be affected by independent influences on key audiences and program’s outcomes
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Types of Evaluation
Formative
Before program development/implementation
Pre-testing
Process and Progress
Compares program’s implementation with original plan
Program reach
Progress monitoring/intermediate results
Summative (outcome or impact evaluation in different models)
Program’s impact in relation to program/outcome objectives (behavioral, social, organizational) and long-term communication objectives
Each type corresponds to a specific
research and monitoring phase
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
What’s Being Measured?
Process Evaluation/Intermediate Results
Message delivery: depth/extent, tone, frequency/ repetition, reach
Message retention: at what time intervals after the event?
Participation in an event: influential advocates/ endorsers, target population
Distribution of materials; number of follow up requests
Endorsement from third-party/KOLs
Ability to create alliances
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
What’s Being Measured?
Process Evaluation/Intermediate Results
Media reach: circulation, impressions
Inquiry for information: toll-free #, website visits
Ability to neutralize opposition
Short-term or intermediate changes in knowledge/ attitudes, skills, community participation levels, etc. (progress)
Summative Evaluation (also Outcome or Impact Evaluation)
Changes in attitude, knowledge, social norms, policies, health or social behavior
Refers to outcome and communication objectives
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Summative Evaluation
“Evaluates program’s efficacy in relation to outcome and communication objectives initially established by the program” (R. Schiavo, 2007, p. 326)
Recognizes current emphasis on strategic behavior communications/behavioral impact
Measures extent to which change occurs
“In health communication programs, the primary objective is usually a health-related behavior.”
J. Bertrand, John Hopkins Center for Health
Communication, The Drum Beat, CI, 2005
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Health Outcomes as a Complex and Multidimensional Construct
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013, figure 3.2. © Copyright 2013. All rights reserved.
Evaluating Behavioral and Social Impact
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
Assessing changes in individual, community, or social behavior requires:
Intensive effort in concentrated geographic area/time period
Front/back market research against
Tracking surveys
Other qualitative/quantitative measurements
Significant financial commitment
Only long-term efforts can generate sustainable behavioral and social results!
11/14/2007
*
How We Measure
Qualitative Methods
In-depth interviews with members of intended audiences, program participants, or other key stakeholders before and after the program
Focus groups
Completion of evaluation forms after specific activities (for process evaluation only)
Evidence of endorsement, such as letters of support or actual program participation from key influentials (for formative and process evaluation only)
Panel studies, that is, pre- and post-intervention studies, which involve the same panel of key stakeholders or representatives of key groups and relevant communities
Community dialogue, public consultations, and other participatory research methods
Quantitative Methods
Pre- and post-surveys—to be implemented in-person, online, by telephone or other methods
Tracking surveys, which collect evaluation data at different time intervals
Control and intervention groups in which groups are either randomized or selected in a way that one group will be exposed to the health communication intervention and the other will not
Online surveys and opinion polls – to take the pulse of progress throughout program implementation and also to evaluate summative results
Source: R. Schiavo, Health Communication: From Theory to Practice. Second Edition. San Francisco: Jossey-Bass, 2013. Figure 14.2, p. 423. All rights reserved. Used by permission.
11/14/2007
*
Current Trends in Evaluation
Increasing number of publications/models on this topic
Reflect need for accountability/new parameters established by major donors
Extent of evaluation efforts depending on:
Size of organization
Optimal balance between evaluation needs, cost, other organizational or program-specific factors
Still, major communication programs with national scope should include all types of evaluation
Mixed methodologies preferred to assess “what” and “why”/link intervention to outcomes
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Current Trends in Evaluation
Behavioral indicators need to be strongly linked to overall program goal
“Increased immunization rates predict decreased child mortality. Increasing number of girls in school often cited as predictor of economic progress.”
Rockefeller Foundation Communication and
Social Change Network, CI
Still, fewer than 1/3 of campaigns express goals/ behavioral objectives in measurable terms
The Health Communication Unit, University
of Toronto
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Current Trends in Evaluation
Participatory evaluation
Mutually agreed upon evaluation parameters,
Include communication team, partners, representatives of key groups, communities, donors, other stakeholders
Logic models/other frameworks used to organize different program elements and related outcomes
Comprehensive evaluation reports
Increased focus on quantitative results/ outcomes (e.g., percentage of intended population where change occurred)
Measurement vis-à-vis outcome objectives (behavioral, social and organizational)
Emphasis on behavioral impact at multiple levels of society = social impact
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Looking at Evaluation Models and Tips
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Assessing Changes in Behavioral Intentions and Behavior
Theory of Reasoned Action (Ajzen and Fishbein, 1980)
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
Behavioral beliefs
Normative beliefs
Subjective norms
Behavioral
Intentions
Attitudes toward
behavior
Behavior
11/14/2007
*
Hicdarm and Behavior Adoption
First, we Hear about the new behavior
then, we become Informed about it
and later Convinced that it is worthwhile.
In time,
we make the Decision to do something about our
conviction
and later we take Action on the new behavior.
We next await Re-confirmation that our action was a
good one
and if all is well,
we Maintain the behavior!
Source: Hosein, E (1979) Communication Training Manual for IEC
(Information-Education-Communication) Officers of IPPF member
affiliates in the Caribbean. Presentation at 1979 International
Planned Parenthood Federation meeting, New York, NY. Used by
permission.
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Logic Model and Evaluation Design
Program’s Name:…..
Long-term Problem to Be Addressed:….
Long-term Objectives to be Achieved…
Source: U.S. DHHS Office of Minority Health, 2010
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
Logic Model and Evaluation Design
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
Source: Adapted model from U.S. DHHS Office of Minority Health, 2010
Contributing Factors Strategies and Tactics Outcomes and Impact
From Your Formative Research/Evaluation How the program addresses the health issue/attempts to achieve outcome objectives (e.g. via community engagement, celebrity engagement, etc.) What kinds of activities the program implemented (tactics)? For example, workshops, PSAs, etc. Short-term/intermediate /outcomes Summative/Behavioral/Long-term outcomes
“Essentials for Excellence” Model
Source: UNICEF, Essentials for Excellence, Nov. 2006, p. 3
Level Research, Monitoring, Evaluation
Program Goal Impact Evaluation
Program Objectives Outcome Evaluation
Behavioral Objectives Behavioral Monitoring, Significant Change Monitoring, Evaluation
Activities (inputs, outputs) Implementation, Monitoring, Pre-testing
11/14/2007
*
Introducing Mutually Agreed Upon Evaluation Parameters
Set specific quantitative/qualitative goals per project prior to implementation
Written assessment upon conclusion
To date, primarily used for process evaluation/ assessment of intermediate changes/progress
Can be used as model to secure consensus on summative evaluation parameters/behavioral indicators
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Process and Progress Evaluation: Establishing Mutually Agreed Parameters. Pre-Implementation Sample Chart
Activity
Healthcare provider workshop series
Radio PSA
Anticipated Quantitative
and Qualitative Results
Reach audience of 30 – 100 depending on venue
80% positive response to evaluations
70-80% speaker endorsement of key messages
Message retention assessed via pre- and post-test market research with target healthcare providers
20-25 million radio impressions over one year period
10,000-15,000 requests for additional information
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. Appendix A. © Copyright 2013. All rights reserved.
11/14/2007
*
Process and Progress Evaluation: Establishing Mutually Agreed Parameters. Pre-implementation Sample
Activity
2. School-based parental outreach
Anticipated Quantitative
and Qualitative Results
Program endorsement by 5-7 schools per city
Message retention assessed using pre- and post-test research with parents in target area
Increase by 15-30% parental awareness and suspicion of disease/ (assessed via pre-/post survey with healthcare providers attending families in target regions)
5,000-10,000 requests for materials fulfilled in each city after initial distribution
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. Appendix A. © Copyright 2013. All rights reserved.
No Matter What Model You Decide to Use Always Consider Evaluation Parameters that Are Inclusive of Vulnerable and Underserved Populations
“Were vulnerable and underserved populations included?
Did they participate in intervention design and evaluation?
Do results among these groups compare to results within the rest of the population?
If these groups were not included, are results relevant?
What issues need to be addressed to ensure inclusion?
What kinds of organizations or stakeholders should be involved to increase cultural competence, use and access?”
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
Developing Evaluation Plans that Suit Your Budget/Program/Organizational Needs
Measurement of outcome indicators (behavioral, social, organizational) a must
Still, small organizations with limited budgets may consider including only 1 or 2 phases of evaluation
All 3 phases should be included for large and/or national programs
In presence of budget/organizational constraints small scale evaluation preferable to poor large scale evaluation
Organizational culture/key stakeholders agreement on parameters critical to successful evaluation
Evaluation budget/plan should be part of original program planning
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
References: R. Schiavo, 2007, 2013; NCI/NHI, 2002
Developing Evaluation Plans that Suit Your Budget/Program/Organizational Needs
Customize evaluation/monitoring tools/methods to meet culturally diverse needs
Consider that some groups may:
Feel uncomfortable providing negative answers in group settings
Have limited reading skills or lack familiarity with questionnaires
Be uneasy speaking with a person of different culture or participate in telephone survey
Develop and use evaluation reports to:
Broadcast results/share lessons learned
Refine program/focus on key strengths
Gain visibility for your program/organization
Advocate for health communication’s worth
Request additional funds
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
References R. Schiavo, 2007, 2013; NCI/NHI, 2002
11/14/2007
*
Developing Evaluation Plans that Suit Your Budget/Program/Organizational Needs
Cost-Effectiveness Tips
Consider using:
General questions from previous evaluation efforts/customize them for specific program
Involve audience members/program partners in data collection
Evaluate cost-effectiveness of different monitoring services/ approaches/team structures
Still, focus on/aim to statistical significance of key results
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
References: R. Schiavo, 2007, 2013; NCI/NHI, 2002
11/14/2007
*
Monitoring: An Essential Element of Program Implementation and Evaluation
Areas of Monitoring
News and trends on disease area/health issue, intended audiences, policies
Ongoing feedback by relevant communities, key groups and stakeholders on program content/vehicles
Process and progress indicators for all program activities/ materials
Collection of summative evaluation data (monitoring behavioral, social and organizational indicators)
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
What is a Monitoring System?
An organized and interrelated set of activities, tools and communities of people that aims at monitoring program and outcome indicators on the basis of a consistent logical framework, work plan and timetable – Schiavo, R.
Relies on in-depth understanding of:
Key questions to be addressed by monitoring efforts
Adequate allocation of human and economic resources
Selection of adequate tools and activities
Advance planning
References: Schiavo, R. 2007, 2013; UNICEF, 2006
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Establishing Sound Monitoring Systems/Teams
Adapted/Summarized from Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013, table 14.4. © Copyright 2013. All rights reserved.
Dedicated staff
members
Community/audience feedback process
Internet/media
analysis/other
special tools/ services
Formal data collection (process, progress, summative)
11/14/2007
*
A Quick Look to the Evaluation Look to the Evaluation of New Media-Based Interventions
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
How is Measurement Being Done? Evaluation of New Media-Based Interventions
Most evaluation efforts still focus on:
Audience exposure/tracking program reach (e.g. number of new or recurring visitors, number of hits, page views, downloads, etc.)
Increased message dissemination/outreach
Increased emphasis on evaluation
Still, most documented case studies on text messaging/mobile use only
Several organizations looking at evaluating awareness and behavioral results of existing/on-going programs
Avatars being used to create proof of concept on potential behavioral impact of new media (e.g. CDC and WhyVille)
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
“Verb [a CDC-campaign] was able to significantly prolong the life of its campaign after the campaign budget ended, as the ads posted on YouTube and its website continued to receive many downloads and views after the campaign officially ended” – Abroms, Schiavo and Lefebvre, 2008; Hunman, 2008
How is Measurement Being Done?
Evaluation of New Media-Based Interventions
Whether online of offline tracking surveys still the best way to assess changes in awareness, knowledge, skills and most importantly behavior
Baseline data should be collected online or offline prior to the introduction of new tools/initiatives
Online efficacy testing methodologies increasingly available to asses internet- and new media-based interventions; still, offline survey/panel studies provide valuable alternatives
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
How is Measurement Being Done?
Evaluation of New Media-Based Interventions
Web-based panels increasingly used to evaluate effectiveness of new media-based interventions
Knowledge Networks – (offer a probability-based sample; geographical samples; laptops provided to users who have not access to technology)
Harris Interactive – (offer large samples of individuals “who have opted in their panel” as part of Harris Poll Online Members)
E-Rewards () and MySurvey () – opinion panels used primarily for commercial marketing purposes
Option to create own sample by using Internet-based tools
(e.g., Survey Monkey – )
Avatars/virtual worlds may be useful tools to measure changes in message relevance/retention/awareness/online efficacy (e.g. CDC and WhyVille)
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Evaluation of New Media-Based Interventions
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013, figure 14.3. © Copyright 2013. All rights reserved.
Assessing Intermediate Steps to Behavioral Change by Using Virtual World’s Tools
Figure 14.3 – Flu Vaccine Campaign 2009 in WhyVille
Source: Flu Vaccine Campaign 2009 in Whyville. www.whyville.net/smmk/nice?source=cdc. Copyright © 2009 Numedeon, Inc. Used by permission.
*
Evaluation of New Media-Based Interventions
Key Trends and Needs
Increased emphasis on evaluation
Most case studies on text messaging/mobile use
Several theories to model/summarize new media’s influence
Need to create:
Proof of concept on:
Potential behavioral impact of new media and/or impact on intermediate steps towards behavior change
Most evidence still as applied to chronic disease management and treatment/copying skills
Link online participation, awareness and behavioral intentions to actual offline behavior
Optimize use of new media; clarify expectations for each specific new media tool
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
*
*
Linking Outcomes to Strategic Communication Programs
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
“Please remember that communications programs take place in a real-world setting, where there are many other influences on your target audiences. In many cases, it may be impossible to isolate the effect of your program on behavioral change from other concomitant activities and/or influences.”
NIH/NCI – Pink Book, 2002
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Linking Outcomes to Strategic Communication Programs
Key Factors
Consistent model for program development, implementation and evaluation1
Well-defined and measurable outcome and communication objectives1
Adequate mix of quantifiable parameters and qualitative analyses1
Intermediate steps to assess progress and process-related results/early changes1
Sound monitoring system1,2/dedicated monitoring team1
Collect data at different stages (baseline, during, end of program)1,2
References: 1) Schiavo,, 2007; 2) UNICEF, Essentials for Excellence, Nov. 2006
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Linking Outcomes to Strategic Communication Programs
Refrain from attempting to link specific activities/materials to long-term changes in knowledge, attitudes, skills or behavior1
“Change does not often occur as a result of just one specific activity3”
Contribution analysis2
Consider plausible alternatives/if existent, factor them in evaluation report2
Adequate funds/cost-effective solutions/early budget allocation1
Sources: 1) Schiavo, 2007; 2) UNICEF, Nov. 2006; 3) NCI/NIH, Making Health Communication, 2002.
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Linking Outcomes to Strategic Communication Programs
“Outcomes appeared at appropriate time after efforts began
Outcomes faded when efforts stopped
Only those outcomes appeared that you should have affected
Outcomes appeared only where/when program was active
Biggest outcomes appeared where/when you did the most
No plausible alternatives for outcomes, or if there are, you have recognized them/ objectively accounted for them in evaluation conclusions”
Source: UNICEF, Essentials for Excellence, Nov. 2006, p. 15
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
11/14/2007
*
Evaluating Health Communication Programs
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. © Copyright 2013. All rights reserved.
The evaluation of health communications programs is a much debated, evolving topic
Recent trends and best practices in evaluating health communication programs include:
Mutually Agreed Upon Parameters
Use of Consistent Models/Design/Logic Models
Participatory Evaluation/Processes
Integration of Vulnerable and Underserved Populations
Mixed Methodologies
Focus on Behavioral and Social Impact/Use of Communication Models and Theories that Support such Behavioral and Social Behavior/Impact Focus
11/14/2007
*
Evaluating Outcomes of Health Communication Interventions
This presentation is part of the instructor’s supplement for Schiavo, R. Health Communication: From Theory to Practice. Second Edition. San Francisco: Jossey-Bass, an imprint of Wiley. The instructor’s supplement is stored within a password-protected webpage for instructors. Copyrights @2013 by Renata Schiavo. All rights reserved
Schiavo, R. Health Communication: From Theory to Practice, San Francisco: Jossey-Bass, Second Edition, 2013. All rights reserved.
*
*