Discussion: Health First Physician Group EHR Implementation Evaluation Plan

Discussion: Health First Physician Group EHR Implementation Evaluation Plan ORDER NOW FOR CUSTOMIZED AND ORIGINAL ESSAY PAPERS ON Discussion: Health First Physician Group EHR Implementation Evaluation Plan Instructions: Building upon your reading and assignment in Module Three and your M4 reading of the Agency for Healthcare Research and Quality (AHRQ): Health Information Technology Evaluation Toolkit 2009 Update https://healthit.ahrq.gov/sites/default/files/docs/page/Evaluation%20Toolkit%20Revised%20Version.pdf , you will compose an Evaluation Plan for Health First Physician Group’s EHR Implementation.Discussion: Health First Physician Group EHR Implementation Evaluation Plan Read pages 1-16. Use this template for your Evaluation Plan of Health First Physician Group’s EHR Implementation. List three or more items for each step I through X and XII –XIV in the Evaluation Plan. Review Section II : Examples of Measures That May Be Used to Evaluate Your Project. This will provide ideas about measures to include in your Evaluation Plan. Submit your plan as an attachment with the following naming convention: Lastname.FirstName.EvaluationPlan ahrq_evaluation_toolkit_2009__1_.docx This document is in the public domain and may be used and reprinted without permission except those copyrighted materials that are clearly noted in the document. Further reproduction of those copyrighted materials is prohibited without the specific permission of copyright holders. Suggested Citation: Cusack CM, Byrne C, Hook JM, McGowan J, Poon EG, Zafar A. Health InformationTechnology Evaluation Toolkit: 2009 Update (Prepared for the AHRQ National Resource Center for Health Information Technology under Contract No. 290-04-0016.) AHRQ Publication No. 09-0083-EF. Rockville, MD: Agency for Healthcare Research and Quality. June 2009. Acknowledgments The authors would like to thank numerous members of the AHRQ National Resource Center’s Value and Evaluation Team for their invaluable input and feedback: Davis Bu, M.D., M.A.(Center for IT Leadership); Karen Cheung, M.P.H. (National Opinion Resource Center); DanGaylin, M.P.A. (National Opinion Resource Center); Julie McGowan, Ph.D. (Indiana UniversitySchool of Medicine); Adil Moiduddin, M.P.P. (National Opinion Resource Center); AnitaSamarth (eHealth Initiative); Jan Walker, R.N., M.B.A. (Center for IT Leadership); and Atif Zafar, M.D. (Indiana University School of Medicine). Thank you also to Mary Darby, Burness Communications , for editorial review . The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services. iContents Introduction ……………………………………………………………………………………………………. 1 Section I: Developing an Evaluation Plan ……………………………………………………… 3 Discussion: Health First Physician Group EHR Implementation Evaluation Plan Discussion: Health First Physician Group EHR Implementation Evaluation Plan Instructions: Building upon your reading and assignment in Module Three and your M4 reading of the Agency for Healthcare Research and Quality (AHRQ): Health Information Technology Evaluation Toolkit 2009 Update https://healthit.ahrq.gov/sites/default/files/docs/page/Evaluation%20Toolkit%20Revised%20Version.pdf , you will compose an Evaluation Plan for Health First Physician Group’s EHR Implementation.Discussion: Health First Physician Group EHR Implementation Evaluation Plan Read pages 1-16. Use this template for your Evaluation Plan of Health First Physician Group’s EHR Implementation. List three or more items for each step I through X and XII –XIV in the Evaluation Plan. Review Section II : Examples of Measures That May Be Used to Evaluate Your Project. This will provide ideas about measures to include in your Evaluation Plan. Submit your plan as an attachment with the following naming convention: Lastname.FirstName.EvaluationPlan ahrq_evaluation_toolkit_2009__1_.docx This document is in the public domain and may be used and reprinted without permission except those copyrighted materials that are clearly noted in the document. Further reproduction of those copyrighted materials is prohibited without the specific permission of copyright holders. Suggested Citation: Cusack CM, Byrne C, Hook JM, McGowan J, Poon EG, Zafar A. Health Information Technology Evaluation Toolkit: 2009 Update (Prepared for the AHRQ National Resource Center for Health Information Technology under Contract No. 290-04-0016.) AHRQ Publication No. 09-0083-EF. Rockville, MD: Agency for Healthcare Research and Quality. June 2009. Acknowledgments The authors would like to thank numerous members of the AHRQ National Resource Center’s Value and Evaluation Team for their invaluable input and feedback: Davis Bu, M.D., M.A. (Center for IT Leadership); Karen Cheung, M.P.H. (National Opinion Resource Center); Dan Gaylin, M.P.A. (National Opinion Resource Center); Julie McGowan, Ph.D. (Indiana University School of Medicine); Adil Moiduddin, M.P.P. (National Opinion Resource Center); Anita Samarth (eHealth Initiative); Jan Walker, R.N., M.B.A. (Center for IT Leadership); and Atif Zafar, M.D. (Indiana University School of Medicine). Thank you also to Mary Darby, Burness Communications , for editorial review . The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services. Discussion: Health First Physician Group EHR Implementation Evaluation Plan i Contents Introduction ……………………………………………………………………………………………………. 1 Section I: Developing an Evaluation Plan ……………………………………………………… 3 Develop Brief Project Description ………………………………………………………3 Determine Project Goals ……………………………………………………………………3 Set Evaluation Goals …………………………………………………………………………4 Choose Evaluation Measures ……………………………………………………………..4 Consider Both Quantitative and Qualitative Measures …………………………..5 Consider Ongoing Evaluation of Barriers, Facilitators, and Lessons Learned …………………………………………………………………………………………..7VII.Search for Other Easily Accessible Measures ………………………………………7 Consider Project Impacts on Potential Measures………………………………….9 Rate Your Chosen Measures in Order of Importance to Your Stakeholders10 Determine Which Measurements Are Feasible …………………………………..10 Determine Your Sample Size……………………………………………………………11 Rank Your Choices on Both Importance And Feasibility …………………….12 Choose the Measures You Want To Evaluate …………………………………….13 Determine Your Study Design ………………………………………………………….13 Consider the Impact of Study Design on Relative Cost And Feasibility ..15 Choose Your Final Measures ……………………………………………………………17 Draft Your Plan Around Each Measure ……………………………………………..19 Write Your Evaluation Plan ……………………………………………………………..20 Section II: Examples of Measures That May Be Used to Evaluate Your Project . 21 Section III: Examples of Projects…………………………………………………………………… 42 Appendixes Appendix A: Sample Size Example ………………………………………………………………….. 56 Appendix B: Health IT Evaluation Resources …………………………………………………… 58 Appendix C: Statistics Resources……………………………………………………………………. 59 ii Introduction We are pleased to present this updated version of the Agency for Healthcare Research and Quality (AHRQ) National Resource Center for Health Information Technology (NRC) Evaluation Toolkit. This toolkit provides step-by-step guidance for project teams who are developing evaluation plans for their health information technology (health IT) projects. You might ask: “Why evaluate?” For years, health IT has been implemented with the goals of improving clinical care processes, health care quality, and patient safety, without questioning the evidence base behind the true impact of these systems. In short, these systems were implemented because they were viewed as the right thing to do. In the early days of health IT implementation, evaluations took a back seat to project work and frequently were not performed at all, at a tremendous loss to the health IT field. Imagine how much easier it would be for you to implement your project if you had solid cost and impact data at your fingertips. Health IT projects require large investments, and, increasingly, stakeholders are demanding information about both the actual and future value of these projects. As a result, we as a field are moving away from talking about theoretical value, to a place where we measure real value. We have reached a point where isolated studies and anecdotal evidence are not enough – not for our stakeholders, nor for the health care community at large. Evaluations must be viewed as an integral piece of every project, not as an afterthought. It is difficult to predict a project’s impact, or even to determine impact once a project is completed. Evaluations allow us to analyze our predictions about our projects and to understand what has worked and what has not. Lessons learned from evaluations help everyone involved in health IT implementation and adoption improve upon what they are doing. In addition, evaluations help justify investment in health IT projects by demonstrating project impacts. This is exactly the type of information needed to convert late adopters and others resistant to health IT. We can also share such information with our communities, raising awareness of efforts in the health IT field on behalf of patient safety and increasing quality of care. Thus, the question posed today is no longer why do we do evaluations but how do we do them? This toolkit will help assist you through the process of planning an evaluation. Section I walks you and your team step-by-step through the process of determining the goals of your project, what is important to your stakeholders, what needs to be measured to satisfy stakeholders, what is realistic and feasible to measure, and how to measure these items. Section II includes a list of measures that you may use to evaluate your project. In this latest version, new measures have been added to each of the domains, and a new domain has been added around quality measures. For each domain, we include a table of possible measures, suggested data sources, cost considerations, potential risks, and general notes. A new column has been added to this updated version of the toolkit, with links to sources that expand on how these measures can be evaluated and with references in the literature. Discussion: Health First Physician Group EHR Implementation Evaluation Plan Section III contains examples of a range of implementation projects with suggested evaluation methodologies for each. In this latest version, two examples have been added on computerized provider order entry (CPOE) and picture archiving and communication systems (PACS). We invite and encourage your feedback on the content, organization, and usefulness of this toolkit as we continue to expand and improve it. Please send your comments or questions about the evaluation toolkit or the National Resource Center to [email protected] . Discussion: Health First Physician Group EHR Implementation Evaluation Plan Section I: Developing an Evaluation Plan I. Develop Brief Project Description This may come straight out of your project plan or proposal. II. Determine Project Goals What does your team hope to gain from this implementation? What are the goals of your stakeholders (CEO, CMO, CFO, clinicians, patients, and so on) for this project? What needs to happen for the project to be deemed a success by your stakeholders? Example: To improve patient safety; to improve the financial position of the hospital; to be seen by our patients as making patient safety an organizational priority. III. Set Evaluation Goals Who is the audience for your evaluation? Do you intend to prepare a report for your stakeholders? Are you required to prepare a report for your funders? Will you use the evaluation to convince late adopters of the value of your implementation? To share lessons learned? To demonstrate the project’s return on investment? To improve your standing and competitive edge in your community? Or are your goals more external? Would you like to share your experiences with a wider audience and publish your findings? If you plan to publish your findings, this may affect your approach to your evaluation. Example: To prepare a report for the stakeholders and funders of the project. IV. Choose Evaluation Measures Take a good look at your project goals. What needs to be measured in order to demonstrate that the project has met those goals? Brainstorm with your team on everything that could be measured, without regard to feasibility. Section II provides a wide range of potential measures in the following categories: Clinical Outcomes Measures Clinical Process Measures Provider Adoption and Attitudes Measures Patient Adoption, Knowledge, and Attitudes Measures Workflow Impact Measures Financial Impact Measures Your team might find it helpful to break down your measures in similar categories. Keep in mind that measures should map back to your original project goals, and that they may include both quantitative and qualitative data. Example: (1) Goal: To improve patient safety. Measurement: The number of preventable adverse drug events is reduced post-implementation. (2) Goal: To improve the hospital’s financial position. Measurement: The number of claims rejected is reduced post-implementation. (3) Goal: To be seen by our patients as making patient safety an organizational priority. Measurement: In patient surveys, patients answer “yes” to the question, “Do you believe this hospital takes your safety seriously?” V. Consider Both Quantitative and Qualitative Measures Many people feel more comfortable in the realm of numbers and, as a result, frequently design their evaluations solely around quantitative data. But this approach provides only a partial picture of your project. Quantitative data can lead to conclusions about your project that miss the larger picture. For example: A hospital implements a new clinical reminder system with the goal of increasing compliance with health maintenance recommendations. An evaluation study is devised to measure the percentage change in the number of patients discharged from the facility who receive influenza vaccines, as recommended. The study is carried out, and, to the disappointment of the research team, the rates of vaccinated patients discharged pre- and post-implementation do not change. The team concludes that their implementation goals have not been met, and that the money spent on the system was a poor investment. But a qualitative study of the behaviors of the clinicians using the new system would have reached different conclusions. In this scenario, the qualitative study reveals that clinicians, bombarded with a number of alerts and health maintenance reminders, click through the alerts without reading them. The influenza vaccine reminders are not read; thus the rates of influenza vaccination remain unchanged. The study also notes that a significant number of clinicians are distracted by and frustrated with the frequent alerts generated by the new system, with no way to distinguish the more important alerts from the less important ones. In addition, some clinicians are unaware of the evidence supporting this vaccine reminder and of the financial (pay-for-performance) implications for the hospital if too few patients receive this vaccine. One clinician had the idea that the vaccine reminder could be added to the common admission order sets. These findings could be used to refocus the design, education, and implementation efforts for this intervention. However, lacking a qualitative evaluation, these insights are lost on the project team. Qualitative studies add another important dimension to an evaluation study: they allow evaluators to understand how users interact with a new system. In addition, qualitative studies speak to a larger audience because they generally are easier to understand than quantitative studies. They often generate anecdotes and stories that resonate with audiences. Therefore, it is important to consider both quantitative and qualitative data in your evaluation plan. Please add any qualitative measures you would like to consider. The National Resource Center has developed a Compendium of Health IT Surveys that may be found on the NRC Web site at Health IT Survey Compendium. This tool allows a user to search for publically available surveys by survey type, technology, care setting, and targeted respondent. These surveys can then be used as is, or can be modified to suit a user’s needs. VI. Consider Ongoing Evaluation of Barriers, Facilitators, and Lessons Learned Lessons learned are important measures of your project and typically are captured using qualitative techniques. These lessons may reflect the facilitators and barriers you encountered at various phases of your project. Barriers may be organizational, financial, or legal, among many other areas. Facilitators might include strong leadership, training, and community buy-in. This type of information is extremely valuable not only to you but also to others undertaking similar projects. In formulating a plan for capturing this information, consider scheduling regular meetings with your project team to discuss the issues at hand openly and to record these discussions. In addition, you could conduct focus groups with appropriate individuals to capture this information more formally. For example, you could ask nurses who are using a new technology about what has gone well, what has gone poorly, and what the unexpected consequences of the project have been. Another way to capture valuable lessons learned is to conduct real-time observations on how users interact with the new technology. Consider how you could incorporate these analysis techniques into your evaluation plan. Clearly state what you want to learn, how you plan to collect the necessary data, and how you would analyze the data. VII. Search for Other Easily Accessible Measures Hospitals collect a tremendous amount of data for multiple purposes: to satisfy various Federal and State requirements, to conduct ongoing quality assurance evaluations, and to measure patient and staff satisfaction. Therefore, there are teams within your facility already collecting data that might be useful to you. Reach out to these groups to learn what information they are currently collecting and to determine whether those data can be used as an evaluation measure. In addition, contact the various departments in your facility to learn the reporting capabilities of their current software programs as well as current data collection methods. There may be opportunities to leverage these reporting capabilities and data collection methods for your project. For example, does the billing department already measure the number of claims rejected? Is there a team already abstracting charts for information that your team would like to examine? Could your team piggy-back with another group to abstract a bit of additional information? Are there useful measurements that could be taken from existing reports? Likewise, you may find that activities you are planning as part of your evaluation would be helpful to other teams within your facility. Cooperation in these activities can increase goodwill on both sides. Section II outlines several potential measures and provides sources where you may find those measures. Example: The finance department’s billing system can report the number of emergency department encounters that are coded as levels I, II, III, IV, and V. These reports are simple to run, and the finance department is willing to run them for you. You already know that many visits are downcoded because a visit was not sufficiently documented – an oversight that can lead to large revenue losses. A new evaluation measure is added to determine whether the new implementation improves documentation so that visits are coded appropriately and revenues are increased. VIII. Consider Project Impacts on Potential Measures A project may have many impacts on a facility, but often these impacts depend on where the project is implemented – for example, across groups of hospitals versus across a single facility versus within a single department. In addition, impacts may vary according to the group that is using a new technology – for example, all facility clinicians versus nurses only. Consider the potential measures on your list and how your project might impact those measures. You may find that this exercise eliminates some measures from your list if you are trying to measure outcomes that will not be impacted by your project. IX. Rate Your Chosen Measures in Order of Importance to Your Stakeholders Now that your team has a list of measures to evaluate, rate each measure in order of importance to your stakeholders, i.e., your CEO, clinicians, or patients, and so on You could use a scale such as: 1 = Very Important, 2 = Moderately Important, 3 = Not Important. This will help you begin to filter out those measures that are interesting to you but will not provide you with information of interest to your stakeholders. Very Important:____________________________________________________ ____________________________________________________________________ Moderately Important:_______________________________________________ ____________________________________________________________________ Not Important:_____________________________________________________ ____________________________________________________________________ X. Determine Which Measurements Are Feasible Now examine your list to determine which measures are feasible for you to measure. Be realistic about the resources available to you. Teams frequently are forced to abandon evaluation projects that are labor-intensive and expensive. Instead, focus on what is achievable and on what needs to be measured to determine whether your implementation has met its goals. For example, you might want to know whether your implementation reduces adverse drug events (ADEs). While this is a terrific evaluation project, if you have neither the money nor the individuals needed for chart abstraction, the project will likely fail. Keep focused on what can be achieved. Again, you can use a ranking scale: 1 = Feasible, 2 = Feasible with Moderate Effort, 3 = Not Feasible. Feasible:__________________________________________________________ ____________________________________________________________________ Moderate Effort:___________________________________________________ ____________________________________________________________________ Not Feasible:_______________________________________________________ ____________________________________________________________________ XI. Determine Your Sample Size A second, extremely important, facet of feasibility is sample size. An evaluation effort can hinge on the number of observations planned or on the frequency of events to be observed. The less frequently the event occurs, the less feasible the planned measure becomes. If a measurement requires a large amount of resources—for example, to directly observe clinicians at work or to conduct manual chart review—or if you are observing very rare events, such as patient deaths, your plan may not be feasible at all. In planning how to study your measure, determine the number of observations you will need to make. Generally, you need enough observations to feel confident about the conclusions you want to draw from the data collected. If you have never estimated a sample size, you should consult a statistician to help you do this correctly or utilize the resources on the AHRQ NRC Web site. Appendix A offers a hypothetical example of determining sample size. Estimate the number of observations you will need for each measure. You may find that this exercise eliminates further measures from being feasible. XII. Rank Your Choices on Both Importance And Feasibility Place your remaining measures into the appropriate box in the grid below. Feasibility Scale 1- Feasible 2- Moderate Effort 3- Not Feasible 1- Very Important (1) (2) 2 Moderately Important (3) (4) 3- Not Important (5) Those measures that fall within the green zone (Most important, Most Feasible) are ones you should definitely undertake; the measures in the yellow zones are ones you can undertake in the order listed; and those measures in the red zone should be avoided. XIII. Choose the Measures You Want To Evaluate You now have a list of measures ranked by importance and feasibility. Narrow that list down to four or five primary measures. If you want to evaluate other measures and you believe that you will have the required resources available to you, list those as secondary measures. XIV. Determine Your Study Design Now that you know which measures you are going to undertake, consider the study design you will use. Listed below are the types of study designs that may be used in your evaluation. Remember that each type of design has attributes of “timing” and “data collection strategy.” Timing can be either retrospective, looking at data from the past, or prospective, looking at new data as it is collected. The data collection strategies include chart reviews, interviews (phone, inperson), focus groups, data mining from electronic databases, observational data collection (time-motion studies), randomized control trials (RCTs), case-control data collection, cohort data collection, automatic data collection (from EMRs), and expert-reviews. This is by no means a substitute for hands-on guidance from a trained statistician. It is only meant to be a ten-thousand foot view of evaluation methods. Below depicts one way of organizing these types of studies: Retrospective Studies Data Collection Strategies Manual Chart Review Electronic Data Mining of EMR/Registry Data Instrument the EMR/Registry (Real-Time Data Collection) iv. Surveys (Paper/Electronic) v. Expert Review vi. Phone Interview vii. Focus Group Study Designs Case Series Case Control Study Prospective Studies Data Collection Strategies Manual Chart Review Electronic Data Mining of EMR/Registry Data Instrument the EMR/Registry (Real-Time Data Collection) iv. Surveys (Paper/Electronic) v. Expert Review vi. Phone Interview vii. Focus Group viii. Direct Observation Study Designs Randomized Control Trial (RCT) Time-Motion Study iii. Pre-Post Study Meta-Analysis Use this table to organize the studies as follows. The shaded areas indicate which strategy fits which design: Data Collection Strategies Types of Study Designs Case-Control RCT Time-Motion Pre-Post Manual Chart Review Electronic Data Mining of EMR/ Registry Data Instrument the EMR/Registry Surveys (Paper/Electronic) Expert Review Phone Interview Focus Group Direct Observation Data Sources – As you think through your study design, you will need to consider where you will obtain your data. Potential sources of data include: Study Databases (Data entered from surveys, focus groups, time-motion studies, and so on) Paper Charts Electronic Data Repositories and EMR databases Lab System Pharmacy System iii. Billing System iv. Registration System Radiology Information System vi. Pathology Information System vii. Health Information Exchange viii. Personal Health Record ix. EMR data (ICD/Procedures) x. Administrative Pharmacy Logs Disease Registries Prescription Review Databases Direct Observation Databases Real-Time Capture from Medical Devices (Barcoders, and so on) Hospital Quality Control Program (Hospital may already be collecting this information for quality reporting.) Consider the Impact of Study Design on Relative Cos t And Feasibility How you have chosen to design your study will impact the feasibility of evaluating a given measure in terms of both the relative cost and the challenges you are likely to encounter. Below we list known caveats around study methodologies and their relative cost considerations, as well as alert you to possible solutions. You may find additional measures you will want to drop from your evaluation plan once you carefully consider these issues. Appendix B includes more resources on Health IT evaluation. Developing your own survey can be time consuming. If you are conducting randomized trials or other rigorous evaluations, you also will need to validate the survey, especially if it is scored, which can add additional time and expense. Some resources on survey design can be found here: Doyle JK. Introduction to survey methodology and design . In: Woods DW. Handbook for IQP advisors and students. Chap. 10. Worcester, MA: Worcester Polytechnic Institute; 2006. AHRQ National Resource Center for Health IT. Health IT survey compendium . California Health Interview Survey. Survey design and methods . Hinkin TR. A brief tutorial on the development of measures for use in survey questionnaires . Organizational Research Methods 1998;1(1):104-21. Focus groups require planning and the logistics can become complicated when busy stakeholders are asked to convene. The methodology for data analysis from focus groups requires the expertise of a qualitative researcher to analyze free-text narratives for themes and common principles. This can also increase the cost of your evaluation quickly. Iowa State University. Focus group fundamentals . Methodology Brief ( PM 1969b) 2004 May. Kitzinger, J. Qualitative research: introducing focus groups . BMJ 1995 Jul 29;311(7000):299-302. Robert Woods Johnson Foundation. Focus groups. Dawson S, Manderson L, Tallo VL. A manual for the use of focus groups (Methods for social research in disease) . Boston, MA: International Nutrition Foundation for Developing Countries; 1993. Manual chart reviews are time consuming and expensive, depending on how many charts you need to review or how many data elements are abstracted. Common pitfalls with chart reviews include unintentional data omission, data entry problems, or the chart itself may be incomplete or have missing information. In addition, reviewers can fatigue easily from the tediousness of the work. Some prospective studies can be done fairly efficiently and quickly. For example, timemotion studies (also known as work-sampling or observational studies) can be quickly performed by motivated research assistants or students at reasonable costs. However, these studies require the development of a list of tasks that the subjects will perform and also require that you have a data collection tool (personal digital assistant-based timer tool, paper-based tool, and so on) where you can record the times for the completion of each task. One could also automate the process by directly “instrumenting” an EMR, meaning specific programming is added to an EMR to capture data. For example, if evaluators want to evaluate the “usefulness of an alert,” programming is added to automatically track every time an alert is fired and every time that alert is followed. In another example, if evaluators want to capture use of e-prescribing, the system will automatically track and aggregate the number of times users prescribe medications electronically. Finkler SA, Knickman JR, Hendrickson G, Lipkin M Jr, Thompson WG. A comparison of work-sampling and time-motion techniques for studies in health services research . Health Serv Res 1993 Dec;28(5):577-97. Caughey MR, Chang BL. Computerized data collection: example of a time motion study. West J Nurs Res 1998 Apr;20(2):251-6 . Other types of prospective studies (randomized controlled trials) and before-after type observational studies are more complicated and expensive. They require modeling of the outcome variables using advanced statistical techniques (generalized linear models, logistic regression, analysis of variance (ANOVA), and so on). While they may provide t